# embeddings_queue

created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0dedac0c-f67a-461c-81bd-2e62400d6045
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0dedac0c-f67a-461c-81bd-2e62400d6045\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cabd36d-6e46-467d-a775-4d9627eb6fff\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee8ef11d-f246-43c1-85c6-bc4edccbb1f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb1e8c8c2c5bec55ae22dc6ecb3ccb784c51d2c593b1464d455ae73ef3fff809\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "ref_doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "chroma:document": "# Financial Document Analysis with LlamaIndex\n\nIn this example notebook, we showcase how to perform financial analysis over [**10-K**](https://en.wikipedia.org/wiki/Form_10-K) documents with the [**LlamaIndex**](https://gpt-index.readthedocs.io/en/latest/) framework with just a few lines of code.\n\n## Notebook Outline\n* [Introduction](#Introduction)\n* [Setup](#Setup)\n* [Data Loading & Indexing](#Data-Loading-and-Indexing)\n* [Simple QA](#Simple-QA)\n* [Advanced QA - Compare and Contrast](#Advanced-QA---Compare-and-Contrast)\n\n\n## Introduction\n\n### LLamaIndex\n[LlamaIndex](https://gpt-index.readthedocs.io/en/latest/) is a data framework for LLM applications.\nYou can get started with just a few lines of code and build a retrieval-augmented generation (RAG) system in minutes.\nFor more advanced users, LlamaIndex offers a rich toolkit for ingesting and indexing your data, modules for retrieval and re-ranking, and composable components for building custom query engines.\n\nSee [full documentation](https://gpt-index.readthedocs.io/en/latest/) for more details.\n\n### Financial Analysis over 10-K documents\nA key part of a financial analyst's job is to extract information and synthesize insight from long financial documents.\nA great example is the 10-K form - an annual report required by the U.S. Securities and Exchange Commission (SEC), that gives a comprehensive summary of a company's financial performance.\nThese documents typically run hundred of pages in length, and contain domain-specific terminology that makes it challenging for a layperson to digest quickly.\n\n\nWe showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. \n\n## Setup\n\nTo begin, we need to install the llama-index library\n\n\n```python\n!pip install llama-index pypdf\n```\n\nNow, we import all modules used in this tutorial\n\n\n```python\nfrom langchain import OpenAI\n\nfrom llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex\nfrom llama_index import set_global_service_context\nfrom llama_index.response.pprint_utils import pprint_response\nfrom llama_index.tools import QueryEngineTool, ToolMetadata\nfrom llama_index.query_engine import SubQuestionQueryEngine\n```\n\nBefore we start, we can configure the LLM provider and model that will power our RAG system.  \nHere, we pick `gpt-3.5-turbo-instruct` from OpenAI.  \n\n\n```python\nllm = OpenAI(temperature=0, model_name=\"gpt-3.5-turbo-instruct\", max_tokens=-1)\n```\n\nWe construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.\n\n\n```python\nservice_context = ServiceContext.from_defaults(llm=llm)\nset_global_service_context(service_context=service_context)\n```\n\n## Data Loading and Indexing\n\nNow, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    \nUnder the hood, the PDFs are converted to plain text `Document` objects, separate by page.  \n\n> Note: this operation might take a while to run, since each document is more than 100 pages."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ee8ef11d-f246-43c1-85c6-bc4edccbb1f5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ee8ef11d-f246-43c1-85c6-bc4edccbb1f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cabd36d-6e46-467d-a775-4d9627eb6fff\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0dedac0c-f67a-461c-81bd-2e62400d6045\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2250595aadff75d1139313acaa93d9c62def9b82f2d44fcc485e3f1c3b81773a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a27bfaee-96f7-435f-b96d-2d013dfaea18\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32f63c194005533c3a439a19e2ed545c88309c0df45aa52f20d5c8f903591004\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1576, \"end_char_idx\": 4689, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "ref_doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "chroma:document": "We showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. \n\n## Setup\n\nTo begin, we need to install the llama-index library\n\n\n```python\n!pip install llama-index pypdf\n```\n\nNow, we import all modules used in this tutorial\n\n\n```python\nfrom langchain import OpenAI\n\nfrom llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex\nfrom llama_index import set_global_service_context\nfrom llama_index.response.pprint_utils import pprint_response\nfrom llama_index.tools import QueryEngineTool, ToolMetadata\nfrom llama_index.query_engine import SubQuestionQueryEngine\n```\n\nBefore we start, we can configure the LLM provider and model that will power our RAG system.  \nHere, we pick `gpt-3.5-turbo-instruct` from OpenAI.  \n\n\n```python\nllm = OpenAI(temperature=0, model_name=\"gpt-3.5-turbo-instruct\", max_tokens=-1)\n```\n\nWe construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.\n\n\n```python\nservice_context = ServiceContext.from_defaults(llm=llm)\nset_global_service_context(service_context=service_context)\n```\n\n## Data Loading and Indexing\n\nNow, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    \nUnder the hood, the PDFs are converted to plain text `Document` objects, separate by page.  \n\n> Note: this operation might take a while to run, since each document is more than 100 pages.\n\n\n```python\nlyft_docs = SimpleDirectoryReader(input_files=[\"../data/10k/lyft_2021.pdf\"]).load_data()\nuber_docs = SimpleDirectoryReader(input_files=[\"../data/10k/uber_2021.pdf\"]).load_data()\n```\n\n\n```python\nprint(f'Loaded lyft 10-K with {len(lyft_docs)} pages')\nprint(f'Loaded Uber 10-K with {len(uber_docs)} pages')\n```\n\n    Loaded lyft 10-K with 238 pages\n    Loaded Uber 10-K with 307 pages\n\n\nNow, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  \n\n> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.\n\n\n```python\nlyft_index = VectorStoreIndex.from_documents(lyft_docs)\nuber_index = VectorStoreIndex.from_documents(uber_docs)\n```\n\n## Simple QA\n\nNow we are ready to run some queries against our indices!  \nTo do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.\n\nFor a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.\n\n\n```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a27bfaee-96f7-435f-b96d-2d013dfaea18
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a27bfaee-96f7-435f-b96d-2d013dfaea18\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cabd36d-6e46-467d-a775-4d9627eb6fff\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee8ef11d-f246-43c1-85c6-bc4edccbb1f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d8ae4af8a52ad9359c05738dcf405e241b747d79b5675f755deb5172f8330a94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d5d2afdb820fa2a38202a5e850a5e4ba352e37c2ec00f12ece24f47bf845decb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3126, \"end_char_idx\": 5855, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "ref_doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "chroma:document": "```python\nlyft_docs = SimpleDirectoryReader(input_files=[\"../data/10k/lyft_2021.pdf\"]).load_data()\nuber_docs = SimpleDirectoryReader(input_files=[\"../data/10k/uber_2021.pdf\"]).load_data()\n```\n\n\n```python\nprint(f'Loaded lyft 10-K with {len(lyft_docs)} pages')\nprint(f'Loaded Uber 10-K with {len(uber_docs)} pages')\n```\n\n    Loaded lyft 10-K with 238 pages\n    Loaded Uber 10-K with 307 pages\n\n\nNow, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  \n\n> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.\n\n\n```python\nlyft_index = VectorStoreIndex.from_documents(lyft_docs)\nuber_index = VectorStoreIndex.from_documents(uber_docs)\n```\n\n## Simple QA\n\nNow we are ready to run some queries against our indices!  \nTo do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.\n\nFor a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.\n\n\n```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)\n\n\n\n```python\nresponse = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $17,455 (page 53)\n\n\n## Advanced QA - Compare and Contrast\n\nFor more complex financial analysis, one often needs to reference multiple documents.  \n\nAs a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  \nFor this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.\n\n\n```python\nquery_engine_tools = [\n    QueryEngineTool(\n        query_engine=lyft_engine, \n        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')\n    ),\n    QueryEngineTool(\n        query_engine=uber_engine, \n        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')\n    ),\n]\n\ns_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)\n```\n\nLet's see these queries in action!"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cabd36d-6e46-467d-a775-4d9627eb6fff\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a27bfaee-96f7-435f-b96d-2d013dfaea18\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7054c28ed1def82986d85e6f63a25ef6ca56f8c21d9496e3113491a06f663852\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3cb489ff-b78c-426b-9235-6734f4f0c5ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"51a1b374e2f803bfacc0f73c739c33b50edadbd7cf22bfea926744b34de7080f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4303, \"end_char_idx\": 7987, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "ref_doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "chroma:document": "```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)\n\n\n\n```python\nresponse = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $17,455 (page 53)\n\n\n## Advanced QA - Compare and Contrast\n\nFor more complex financial analysis, one often needs to reference multiple documents.  \n\nAs a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  \nFor this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.\n\n\n```python\nquery_engine_tools = [\n    QueryEngineTool(\n        query_engine=lyft_engine, \n        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')\n    ),\n    QueryEngineTool(\n        query_engine=uber_engine, \n        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')\n    ),\n]\n\ns_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)\n```\n\nLet's see these queries in action!\n\n\n```python\nresponse = await s_engine.aquery('Compare and contrast the customer segments and geographies that grew the fastest')\n```\n\n    Generated 4 sub questions.\n    \u001b[36;1m\u001b[1;3m[uber_10k] Q: What customer segments grew the fastest for Uber\n    \u001b[0m\u001b[36;1m\u001b[1;3m[uber_10k] A: in 2021?\n    \n    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth.\n    \u001b[0m\u001b[33;1m\u001b[1;3m[uber_10k] Q: What geographies grew the fastest for Uber\n    \u001b[0m\u001b[33;1m\u001b[1;3m[uber_10k] A: \n    Based on the context information, it appears that Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.\n    \u001b[0m\u001b[38;5;200m\u001b[1;3m[lyft_10k] Q: What customer segments grew the fastest for Lyft\n    \u001b[0m\u001b[38;5;200m\u001b[1;3m[lyft_10k] A: \n    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them.\n    \u001b[0m\u001b[32;1m\u001b[1;3m[lyft_10k] Q: What geographies grew the fastest for Lyft\n    \u001b[0m\u001b[32;1m\u001b[1;3m[lyft_10k] A: \n    It is not possible to answer this question with the given context information.\n    \u001b[0m"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3cb489ff-b78c-426b-9235-6734f4f0c5ac
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3cb489ff-b78c-426b-9235-6734f4f0c5ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1cabd36d-6e46-467d-a775-4d9627eb6fff\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a71d8cfafbb8a73e35af5b561ae16135a807711966e3e1f2d3170bdca7b40ad3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7990, \"end_char_idx\": 10896, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "ref_doc_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "chroma:document": "```python\nprint(response)\n```\n\n    \n    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth. Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.\n    \n    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.\n    \n    In summary, Uber and Lyft both experienced growth in customer segments related to mobility, couriers, riders, and eaters. Uber experienced the most growth in large metropolitan areas, as well as in suburban and rural areas, and in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain. Lyft experienced the most growth in ridesharing, light vehicles, and public transit. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.\n\n\n\n```python\nresponse = await s_engine.aquery('Compare revenue growth of Uber and Lyft from 2020 to 2021')\n```\n\n    Generated 2 sub questions.\n    \u001b[36;1m\u001b[1;3m[uber_10k] Q: What is the revenue growth of Uber from 2020 to 2021\n    \u001b[0m\u001b[36;1m\u001b[1;3m[uber_10k] A: \n    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis.\n    \u001b[0m\u001b[33;1m\u001b[1;3m[lyft_10k] Q: What is the revenue growth of Lyft from 2020 to 2021\n    \u001b[0m\u001b[33;1m\u001b[1;3m[lyft_10k] A: \n    The revenue growth of Lyft from 2020 to 2021 is 36%, increasing from $2,364,681 thousand to $3,208,323 thousand.\n    \u001b[0m\n\n\n```python\nprint(response)\n```\n\n    \n    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis, while the revenue growth of Lyft from 2020 to 2021 was 36%. This means that Uber had a higher revenue growth than Lyft from 2020 to 2021."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5a5630fa-3a56-4d7b-a773-1b30b9a5cd14
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5a5630fa-3a56-4d7b-a773-1b30b9a5cd14\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d3a645e5-d0d8-44f8-82f6-03f9e2dd4469\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0c4290ce-d0e8-4bc4-8602-de99b47db5fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"363a84282b8220a59fa73aa021e726f76ba41f77e326997513e7208d09cc6883\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3031, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "ref_doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "chroma:document": "# OpenAI API Monitoring with W&B Weave\n\n<img src=\"http://wandb.me/logo-im-png\" width=\"400\" alt=\"Weights & Biases\" />\n\n<!--- @wandbcode{weave_openai_client_qs} -->\n\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/wandb/weave/blob/master/examples/prompts/llm_monitoring/openai_client_quickstart.ipynb\">\n  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n</a>\n\n**Note:** you will need an [OpenAI API key](https://platform.openai.com/account/api-keys) to run this colab.\n\nUse the W&B OpenAI integration to monitor OpenAI API calls and understand how your projects and teams are leveraging LLMs.\nIn this example, we'll generate templated Weave Boards: LLM usage monitoring dashboards which you can explore and customize from the UI.\n\n* automatically track LLM usage and aggregate useful metrics like  cost, latency and throughput across your projects/teams\n* dynamically query and derive insights from the logs of all your OpenAI API calls\n* iterate visually to slice, aggregate, and explore your data; customize panels to focus on interesting patterns; share progress more easily with your team through an interactive dashboard\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/full_board_view.png\">\n\n[Play with a live version of this Weave Board \u2192](http://wandb.me/llm-monitoring-board)\n\n#### New to Weights & Biases? [-> Sign up for an account here <-](https://wandb.ai/site)\n\n# Step 0: Setup\n\nInstall dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.\n\n\n```python\n# if not already installed\n!pip install -qqq weave openai tiktoken wandb\n```\n\n\n```python\nimport wandb\nwandb.login()\n```\n\n\n```python\nimport weave\nimport os\nWANDB_BASE_URL = \"https://api.wandb.ai\"\nos.environ[\"WANDB_BASE_URL\"] = WANDB_BASE_URL\n```\n\n\n```python\n# authenticate with OpenAI\nfrom getpass import getpass\n\nif os.getenv(\"OPENAI_API_KEY\") is None:\n  os.environ[\"OPENAI_API_KEY\"] = getpass(\"Paste your OpenAI key from: https://platform.openai.com/account/api-keys\\n\")\nassert os.getenv(\"OPENAI_API_KEY\", \"\").startswith(\"sk-\"), \"This doesn't look like a valid OpenAI API key\"\nprint(\"OpenAI API key configured\")\n```\n\n# Step 1: Configure data streaming and storage in W&B\n\nSet WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your \"Profile\" and \"Teams\" in the left sidebar.\n\n\n```python\nWB_ENTITY = \"\" # set to your wandb username or team name\nWB_PROJECT = \"weave\" # top-level directory for this work\nSTREAM_NAME = \"openai_logs\" # record table which stores the logs of OpenAI API calls as they stream in\n```\n\n# Step 2: Call init_monitor()\n\nTo start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.\n\nRunning this cell will print out a link to view the current project in the Weave UI."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0c4290ce-d0e8-4bc4-8602-de99b47db5fe
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0c4290ce-d0e8-4bc4-8602-de99b47db5fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d3a645e5-d0d8-44f8-82f6-03f9e2dd4469\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a5630fa-3a56-4d7b-a773-1b30b9a5cd14\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e400e76af28ddd58b63efd214cc2a9578c0592268c9a1578a82675b6c47bd75e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"892d587d-7a09-4806-b017-76b433cdc643\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396f67fe8962b03eee56e69bfd8b79cfbafc12da60f5d3b5f9c70ac405c03544\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1581, \"end_char_idx\": 4562, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "ref_doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "chroma:document": "```python\n# if not already installed\n!pip install -qqq weave openai tiktoken wandb\n```\n\n\n```python\nimport wandb\nwandb.login()\n```\n\n\n```python\nimport weave\nimport os\nWANDB_BASE_URL = \"https://api.wandb.ai\"\nos.environ[\"WANDB_BASE_URL\"] = WANDB_BASE_URL\n```\n\n\n```python\n# authenticate with OpenAI\nfrom getpass import getpass\n\nif os.getenv(\"OPENAI_API_KEY\") is None:\n  os.environ[\"OPENAI_API_KEY\"] = getpass(\"Paste your OpenAI key from: https://platform.openai.com/account/api-keys\\n\")\nassert os.getenv(\"OPENAI_API_KEY\", \"\").startswith(\"sk-\"), \"This doesn't look like a valid OpenAI API key\"\nprint(\"OpenAI API key configured\")\n```\n\n# Step 1: Configure data streaming and storage in W&B\n\nSet WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your \"Profile\" and \"Teams\" in the left sidebar.\n\n\n```python\nWB_ENTITY = \"\" # set to your wandb username or team name\nWB_PROJECT = \"weave\" # top-level directory for this work\nSTREAM_NAME = \"openai_logs\" # record table which stores the logs of OpenAI API calls as they stream in\n```\n\n# Step 2: Call init_monitor()\n\nTo start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.\n\nRunning this cell will print out a link to view the current project in the Weave UI.\n\n\n```python\nfrom weave.monitoring import openai, init_monitor\nm = init_monitor(f\"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}\")\n\n# specifying a single model for simplicity\nOPENAI_MODEL = 'gpt-3.5-turbo'\n\n# prefill with some sample logs\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"hello world!\"}])\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"what is 2+2?\"}])\n```\n\n# Step 3: Preview monitoring dashboard\n\nClick on the link above to preview the data stream, then click \"OpenAI Monitor Board\" in the right sidebar to create a Weave Board for this data stream.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif\" width=75%>\n\n# Step 4: Explore & understand your LLM usage\n\nTo save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \\\"Publish\\\" in the top right.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif\" width=75%>\n\nTo visualize your work in real-time as you iterate, you can:\n* keep the Board open in a separate tab and refresh to view the latest data\n* rename the Board for easier reference at any point and \\\"Publish\\\" that version to share a link with others\n* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai\n* or open a new instance of a Board template to start fresh with all the data accumulated so far"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 892d587d-7a09-4806-b017-76b433cdc643
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"892d587d-7a09-4806-b017-76b433cdc643\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d3a645e5-d0d8-44f8-82f6-03f9e2dd4469\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0c4290ce-d0e8-4bc4-8602-de99b47db5fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1bd4858a35746f66aff2f418828e38692cc442dffaac25318ee8e5173ee52e48\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90a6a3da-aa4d-4547-84dc-ab78c8988ff4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12347373e3f59afb296ebc2c75e1ba11ad2c3e89b670fd2ef0e245d794e5230c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3034, \"end_char_idx\": 6586, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "ref_doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "chroma:document": "```python\nfrom weave.monitoring import openai, init_monitor\nm = init_monitor(f\"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}\")\n\n# specifying a single model for simplicity\nOPENAI_MODEL = 'gpt-3.5-turbo'\n\n# prefill with some sample logs\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"hello world!\"}])\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"what is 2+2?\"}])\n```\n\n# Step 3: Preview monitoring dashboard\n\nClick on the link above to preview the data stream, then click \"OpenAI Monitor Board\" in the right sidebar to create a Weave Board for this data stream.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif\" width=75%>\n\n# Step 4: Explore & understand your LLM usage\n\nTo save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \\\"Publish\\\" in the top right.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif\" width=75%>\n\nTo visualize your work in real-time as you iterate, you can:\n* keep the Board open in a separate tab and refresh to view the latest data\n* rename the Board for easier reference at any point and \\\"Publish\\\" that version to share a link with others\n* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai\n* or open a new instance of a Board template to start fresh with all the data accumulated so far\n\n\nNext we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.\n\n# Examples\n\n## Example 0: Log a prompt and its completion\n\nMonitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.\n\n\n```python\nresponse = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"user\", \"content\": f\"What is the meaning of life, the universe, and everything?\"},\n    ])\nprint(response['choices'][0]['message']['content'])\n```\n\n## Example 1: Track relevant parameters as attributes\n\nFactor out parameters of interest and track them as attributes on the logged record.\nHere we track the \"system prompt\" separately from the \"prompt template\" and the \"equation\" parameter. This time we'll print the full structured response from the ChatCompletion call.\n\n\n```python\nsystem_prompt = \"you always write in bullet points\"\nprompt_template = 'solve the following equation step by step: {equation}'\nparams = {'equation': '4 * (3 - 1)'}\nopenai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n## Example 2: Log an ongoing stream of messages\n\nMonitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 90a6a3da-aa4d-4547-84dc-ab78c8988ff4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"90a6a3da-aa4d-4547-84dc-ab78c8988ff4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d3a645e5-d0d8-44f8-82f6-03f9e2dd4469\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"892d587d-7a09-4806-b017-76b433cdc643\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"98df897b98cad5fcc09ae7ff45d9331b3424820ee8a16e75991fa0ebc0592d15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ee6a028-fdd3-48af-a32a-489d127f2e04\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"34dfec5596283801792b850a60936b8ebce3d278c76c75954c0abbb19eeefdcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4565, \"end_char_idx\": 8061, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "ref_doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "chroma:document": "Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.\n\n# Examples\n\n## Example 0: Log a prompt and its completion\n\nMonitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.\n\n\n```python\nresponse = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"user\", \"content\": f\"What is the meaning of life, the universe, and everything?\"},\n    ])\nprint(response['choices'][0]['message']['content'])\n```\n\n## Example 1: Track relevant parameters as attributes\n\nFactor out parameters of interest and track them as attributes on the logged record.\nHere we track the \"system prompt\" separately from the \"prompt template\" and the \"equation\" parameter. This time we'll print the full structured response from the ChatCompletion call.\n\n\n```python\nsystem_prompt = \"you always write in bullet points\"\nprompt_template = 'solve the following equation step by step: {equation}'\nparams = {'equation': '4 * (3 - 1)'}\nopenai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n## Example 2: Log an ongoing stream of messages\n\nMonitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.\n\n\n```python\nfrom weave.monitoring.openai import message_from_stream\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"system\", \"content\": \"You are a robot and only speak in robot, like beep bloop bop.\"},\n        {\"role\": \"user\", \"content\": \"Tell me a 50-word story.\"},\n    ], stream=True)\nfor s in message_from_stream(r):\n    print(s, end='')\n```\n\n## Example 3: Structure prompt engineering experiments\n\nHere we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.\n\n\n```python\ndef explain_math(system_prompt, prompt_template, params):\n    openai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4ee6a028-fdd3-48af-a32a-489d127f2e04
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4ee6a028-fdd3-48af-a32a-489d127f2e04\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d3a645e5-d0d8-44f8-82f6-03f9e2dd4469\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90a6a3da-aa4d-4547-84dc-ab78c8988ff4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"13686e93f8ba875da4387aca3d715df0cb6b6d4ecf4faefd96b0f1163898213d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6589, \"end_char_idx\": 8659, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "ref_doc_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "chroma:document": "```python\nfrom weave.monitoring.openai import message_from_stream\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"system\", \"content\": \"You are a robot and only speak in robot, like beep bloop bop.\"},\n        {\"role\": \"user\", \"content\": \"Tell me a 50-word story.\"},\n    ], stream=True)\nfor s in message_from_stream(r):\n    print(s, end='')\n```\n\n## Example 3: Structure prompt engineering experiments\n\nHere we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.\n\n\n```python\ndef explain_math(system_prompt, prompt_template, params):\n    openai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n\n```python\n# feel free to substitute your own prompts :)\nsystem_prompts = [\"you're extremely flowery and poetic\", \"you're very direct and precise\", \"balance brevity with insight\"]\nprompt_template = 'explain the solution of the following to a {audience}: {equation}'\nequations = ['x^2 + 4x + 9 = 0', '15 * (2 - 6) / 4']\naudience = [\"new student\", \"math genius\"]\n\nfor system_prompt in system_prompts:\n    for equation in equations:\n        for person in audience:\n            params = {\"equation\" : equation, \"audience\" : person}\n            explain_math(system_prompt, prompt_template, params)\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f0fb5af5-272c-4c30-a875-bfad8fd9199f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f0fb5af5-272c-4c30-a875-bfad8fd9199f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9cf62e7-0c41-4e89-a139-6ca5488c0bc5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2d781cee85ceb8a18b7dbc93b9f0c9e753a9a6314c0b5f46cc41569d0c33ea40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "# How to use the DALL\u00b7E API\n\nThis notebook shows how to use OpenAI's DALL\u00b7E image API endpoints.\n\nThere are three API endpoints:\n- **Generations:** generates an image or images based on an input caption\n- **Edits:** edits or extends an existing image\n- **Variations:** generates variations of an input image\n\n## Setup\n\n- Import the packages you'll need\n- Import your OpenAI API key: You can do this by running \\``export OPENAI_API_KEY=\"your API key\"`\\` in your terminal.\n- Set a directory to save images to\n\n\n```python\n# imports\nfrom openai import OpenAI  # OpenAI Python library to make API calls\nimport requests  # used to download images\nimport os  # used to access filepaths\nfrom PIL import Image  # used to print and edit images\n\n# initialize OpenAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n```python\n# set a directory to save DALL\u00b7E images to\nimage_dir_name = \"images\"\nimage_dir = os.path.join(os.curdir, image_dir_name)\n\n# create the directory if it doesn't yet exist\nif not os.path.isdir(image_dir):\n    os.mkdir(image_dir)\n\n# print the directory to save to\nprint(f\"{image_dir=}\")\n\n```\n\n    image_dir='./images'\n\n\n## Generations\n\nThe generation API endpoint creates an image based on a text prompt. [API Reference](https://platform.openai.com/docs/api-reference/images/create)\n\n**Required inputs:**\n- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters for dall-e-2 and 4000 characters for dall-e-3.\n\n**Optional inputs:**\n- `model` (str): The model to use for image generation. Defaults to dall-e-2\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `quality` (str): The quality of the image that will be generated. hd creates images with finer details and greater consistency across the image. This param is only supported for dall-e-3.\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `size` (str): The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for dall-e-2. Must be one of 1024x1024, 1792x1024, or 1024x1792 for dall-e-3 models. Defaults to \"1024x1024\".\n- `style`(str | null): The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. This param is only supported for dall-e-3.\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b9cf62e7-0c41-4e89-a139-6ca5488c0bc5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b9cf62e7-0c41-4e89-a139-6ca5488c0bc5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0fb5af5-272c-4c30-a875-bfad8fd9199f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c1d2f0a793e18b44a725df682a674e6aafdd589f8085d30ff8e27904aa3b3177\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7bd2a930-b8a3-4efc-80a1-0b915f51b282\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cae7f3a0656f5991322954cdf529746689f54e953920540a1bdb209958abb39d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2740, \"end_char_idx\": 4169, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "```python\n# create an image\n\n# set the prompt\nprompt = \"A cyberpunk monkey hacker dreaming of a beautiful bunch of bananas, digital art\"\n\n# call the OpenAI API\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=prompt,\n    n=1,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(generation_response)\n```\n\n    ImagesResponse(created=1701994117, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-ced13hkOk3lXkccQgW1fAQjm.png?st=2023-12-07T23%3A08%3A37Z&se=2023-12-08T01%3A08%3A37Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A41%3A48Z&ske=2023-12-08T16%3A41%3A48Z&sks=b&skv=2021-08-06&sig=tcD0iyU0ABOvWAKsY89gp5hLVIYkoSXQnrcmH%2Brkric%3D')])\n\n\n\n```python\n# save the image\ngenerated_image_name = \"generated_image.png\"  # any name you like; the filetype should be .png\ngenerated_image_filepath = os.path.join(image_dir, generated_image_name)\ngenerated_image_url = generation_response.data[0].url  # extract image URL from response\ngenerated_image = requests.get(generated_image_url).content  # download the image\n\nwith open(generated_image_filepath, \"wb\") as image_file:\n    image_file.write(generated_image)  # write the image to the file\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7bd2a930-b8a3-4efc-80a1-0b915f51b282
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7bd2a930-b8a3-4efc-80a1-0b915f51b282\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9cf62e7-0c41-4e89-a139-6ca5488c0bc5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ca9863e3282d4ce23b600be97a33d3bae0994bc757d120ace5dda9a37610cb1f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"74a4edab-6576-4453-827e-b5675a2085f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5c5938ad4a3ef80b7ee11a01846346a4617862da6dd403ad94bc1025cc1ca3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3673, \"end_char_idx\": 5363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "```python\n# save the image\ngenerated_image_name = \"generated_image.png\"  # any name you like; the filetype should be .png\ngenerated_image_filepath = os.path.join(image_dir, generated_image_name)\ngenerated_image_url = generation_response.data[0].url  # extract image URL from response\ngenerated_image = requests.get(generated_image_url).content  # download the image\n\nwith open(generated_image_filepath, \"wb\") as image_file:\n    image_file.write(generated_image)  # write the image to the file\n```\n\n\n```python\n# print the image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n```\n\n## Variations\n\nThe variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)\n\nHere we'll generate variations of the image generated above.\n\n**Required inputs:**\n- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.\n\n**Optional inputs:**\n- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 74a4edab-6576-4453-827e-b5675a2085f0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"74a4edab-6576-4453-827e-b5675a2085f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7bd2a930-b8a3-4efc-80a1-0b915f51b282\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"61ddeebe84698449c483f81b29c6a233acb9a00b3dc64cab2474640d833bca38\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"77b745b8-dfcc-4a3d-8c64-c2b685c899a2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"becccdac305bcb78ceefb8359364540a75d0fd484b5c57e223ebb1d766a85edc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4172, \"end_char_idx\": 6795, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "```python\n# print the image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n```\n\n## Variations\n\nThe variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)\n\nHere we'll generate variations of the image generated above.\n\n**Required inputs:**\n- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.\n\n**Optional inputs:**\n- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)\n\n\n\n```python\n# create variations\n\n# call the OpenAI API, using `create_variation` rather than `create`\nvariation_response = client.images.create_variation(\n    image=generated_image,  # generated_image is the image generated above\n    n=2,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(variation_response)\n```\n\n    ImagesResponse(created=1701994139, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-noNRGgwaaotRGIe6Y2GVeSpr.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=ER5RUglhtIk9LWJXw1DsolorT4bnEmFostfnUjY21ns%3D'), Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-oz952tL11FFhf9iXXJVIRUZX.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=99rJOQwDKsfIeerlMXMHholhAhrHfYaQRFJBF8FKv74%3D')])"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 77b745b8-dfcc-4a3d-8c64-c2b685c899a2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"77b745b8-dfcc-4a3d-8c64-c2b685c899a2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"74a4edab-6576-4453-827e-b5675a2085f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b7c14f199b7dfd341139a1866db80829d2e7c7d1c9713c416fe03c40ecd2211e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d146b79fe58b50bc0cf29dbad89ef05d81adc82a9dee84541e043cf912021a08\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6799, \"end_char_idx\": 9501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "```python\n# save the images\nvariation_urls = [datum.url for datum in variation_response.data]  # extract URLs\nvariation_images = [requests.get(url).content for url in variation_urls]  # download images\nvariation_image_names = [f\"variation_image_{i}.png\" for i in range(len(variation_images))]  # create names\nvariation_image_filepaths = [os.path.join(image_dir, name) for name in variation_image_names]  # create filepaths\nfor image, filepath in zip(variation_images, variation_image_filepaths):  # loop through the variations\n    with open(filepath, \"wb\") as image_file:  # open the file\n        image_file.write(image)  # write the image to the file\n```\n\n\n```python\n# print the original image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n# print the new variations\nfor variation_image_filepaths in variation_image_filepaths:\n    print(variation_image_filepaths)\n    display(Image.open(variation_image_filepaths))\n\n```\n\n## Edits\n\nThe edit endpoint uses DALL\u00b7E to generate a specified portion of an existing image. Three inputs are needed: the image to edit, a mask specifying the portion to be regenerated, and a prompt describing the desired image. [API Reference](https://platform.openai.com/docs/api-reference/images/createEdit)\n\n**Required inputs:** \n- `image` (str): The image to edit. Must be a valid PNG file, less than 4MB, and square. If mask is not provided, image must have transparency, which will be used as the mask.\n- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters.\n\n**Optional inputs:**\n- `mask` (file): An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where image should be edited. Must be a valid PNG file, less than 4MB, and have the same dimensions as image.\n- `model` (str): The model to use for edit image. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)\n\n\n### Set Edit Area\n\nAn edit requires a \"mask\" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"77b745b8-dfcc-4a3d-8c64-c2b685c899a2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"039f85b4ba8c1c5af27ed24680cbc2a932b47a226ca57b48a99ec8a540ae6373\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fbd66a5-b6bd-47c8-b7fd-362b05fd18a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9f90653b2f6b2afeecea1e3baea97bdebc00fefe65a847bf3f4757133d59617\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9262, \"end_char_idx\": 11576, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "### Set Edit Area\n\nAn edit requires a \"mask\" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.\n\n\n```python\n# create a mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# set the bottom half to be transparent\nfor x in range(width):\n    for y in range(height // 2, height):  # only loop over the bottom half of the mask\n        # set alpha (A) to zero to turn pixel transparent\n        alpha = 0\n        mask.putpixel((x, y), (0, 0, 0, alpha))\n\n# save the mask\nmask_name = \"bottom_half_mask.png\"\nmask_filepath = os.path.join(image_dir, mask_name)\nmask.save(mask_filepath)\n```\n\n### Perform Edit\n\nNow we supply our image, caption and mask to the API to get 5 examples of edits to our image\n\n\n```python\n# edit an image\n\n# call the OpenAI API\nedit_response = client.images.edit(\n    image=open(generated_image_filepath, \"rb\"),  # from the generation section\n    mask=open(mask_filepath, \"rb\"),  # from right above\n    prompt=prompt,  # from the generation section\n    n=1,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(edit_response)\n```\n\n    ImagesResponse(created=1701994167, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-9UOVGC7wB8MS2Q7Rwgj0fFBq.png?st=2023-12-07T23%3A09%3A27Z&se=2023-12-08T01%3A09%3A27Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A40%3A37Z&ske=2023-12-08T16%3A40%3A37Z&sks=b&skv=2021-08-06&sig=MsRMZ1rN434bVdWr%2B9kIoqu9CIrvZypZBfkQPTOhCl4%3D')])\n\n\n\n```python\n# save the image\nedited_image_name = \"edited_image.png\"  # any name you like; the filetype should be .png\nedited_image_filepath = os.path.join(image_dir, edited_image_name)\nedited_image_url = edit_response.data[0].url  # extract image URL from response\nedited_image = requests.get(edited_image_url).content  # download the image\n\nwith open(edited_image_filepath, \"wb\") as image_file:\n    image_file.write(edited_image)  # write the image to the file\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3fbd66a5-b6bd-47c8-b7fd-362b05fd18a1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3fbd66a5-b6bd-47c8-b7fd-362b05fd18a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"77807684d9069d61decc78a9f4611e6d5830bbcba09168dcae74cff68a3d76cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11113, \"end_char_idx\": 11791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "ref_doc_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "chroma:document": "```python\n# save the image\nedited_image_name = \"edited_image.png\"  # any name you like; the filetype should be .png\nedited_image_filepath = os.path.join(image_dir, edited_image_name)\nedited_image_url = edit_response.data[0].url  # extract image URL from response\nedited_image = requests.get(edited_image_url).content  # download the image\n\nwith open(edited_image_filepath, \"wb\") as image_file:\n    image_file.write(edited_image)  # write the image to the file\n```\n\n\n```python\n# print the original image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n# print edited image\nprint(edited_image_filepath)\ndisplay(Image.open(edited_image_filepath))\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9ef051b8-63d6-42bc-bb14-2f7e3cadb37d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9ef051b8-63d6-42bc-bb14-2f7e3cadb37d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"502db638-9743-42da-b129-274f9d9d7c53\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f872709cd0ad71669adf9a32846c3719ad168a0d9fe6b7f1d34ff216c84207e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "# How to create dynamic masks with DALL\u00b7E and Segment Anything\n\nSegment Anything is a model from Meta that can be used to select portions of images. Combined with DALL\u00b7E's ability to inpaint specified portions of images, you can use Segment Anything to easily select any part of an image you'd like to alter.\n\nIn this notebook, we'll use these tools to become fashion designers and dynamically replace our digital models' outfits with tailored, original creations. The notebook follows this flow:\n- **Setup:** Initialise your libraries and any location directories.\n- **Generate original image:** Make an original image that we'll create dynamic masks from.\n- **Generate mask:** Use Segment Anything to create a dynamic mask.\n- **Create new image:** Generate a new image with the masked area inpainted with a fresh prompt.\n\n## Setup\n\nTo get started we'll need to follow the [instructions](https://github.com/facebookresearch/segment-anything) for using the Segment Anything (SAM) model open-sourced by Meta. As of May 2023, the key steps are:\n- Install [Pytorch](https://pytorch.org/get-started/locally/) (version 1.7+).\n- Install the library using ```pip install git+https://github.com/facebookresearch/segment-anything.git```.\n- Install dependencies using ```pip install opencv-python pycocotools matplotlib onnxruntime onnx```.\n- Download a [model checkpoint](https://github.com/facebookresearch/segment-anything#model-checkpoints) to use (default size is 2.4 GB).\n\n\n\n```python\n!pip install torch torchvision torchaudio\n!pip install git+https://github.com/facebookresearch/segment-anything.git\n!pip install opencv-python pycocotools matplotlib onnxruntime onnx\n!pip install requests\n!pip install openai\n!pip install numpy\n```\n\n\n```python\n!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n```\n\n\n```python\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib import rcParams\nimport numpy as np\nfrom openai import OpenAI\nimport os\nfrom PIL import Image\nimport requests\nfrom segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor\nimport torch\n\n# Set directories for generation images and edit images\nbase_image_dir = os.path.join(\"images\", \"01_generations\")\nmask_dir = os.path.join(\"images\", \"02_masks\")\nedit_image_dir = os.path.join(\"images\", \"03_edits\")\n\n# Point to your downloaded SAM model\nsam_model_filepath = \"./sam_vit_h_4b8939.pth\"\n\n# Initiate SAM model\nsam = sam_model_registry[\"default\"](checkpoint=sam_model_filepath)\n\n# Initiate openAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n## Generate original image\n\nFirst we'll create an original image which we'll generate masks from."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 502db638-9743-42da-b129-274f9d9d7c53
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"502db638-9743-42da-b129-274f9d9d7c53\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ef051b8-63d6-42bc-bb14-2f7e3cadb37d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4ddc53453b171b4f334074341e934edcf7591744a638c0009f1759ad6db9ba3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"53d27639-1aed-4bda-a984-4d46365f1933\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"150cbdb4185cb59868a7c988f47a1ba89266553763a874c355bb162ad93ba927\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1471, \"end_char_idx\": 4084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "```python\n!pip install torch torchvision torchaudio\n!pip install git+https://github.com/facebookresearch/segment-anything.git\n!pip install opencv-python pycocotools matplotlib onnxruntime onnx\n!pip install requests\n!pip install openai\n!pip install numpy\n```\n\n\n```python\n!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n```\n\n\n```python\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib import rcParams\nimport numpy as np\nfrom openai import OpenAI\nimport os\nfrom PIL import Image\nimport requests\nfrom segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor\nimport torch\n\n# Set directories for generation images and edit images\nbase_image_dir = os.path.join(\"images\", \"01_generations\")\nmask_dir = os.path.join(\"images\", \"02_masks\")\nedit_image_dir = os.path.join(\"images\", \"03_edits\")\n\n# Point to your downloaded SAM model\nsam_model_filepath = \"./sam_vit_h_4b8939.pth\"\n\n# Initiate SAM model\nsam = sam_model_registry[\"default\"](checkpoint=sam_model_filepath)\n\n# Initiate openAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n## Generate original image\n\nFirst we'll create an original image which we'll generate masks from.\n\n\n```python\ndef process_dalle_images(response, filename, image_dir):\n    # save the images\n    urls = [datum.url for datum in response.data]  # extract URLs\n    images = [requests.get(url).content for url in urls]  # download images\n    image_names = [f\"{filename}_{i + 1}.png\" for i in range(len(images))]  # create names\n    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths\n    for image, filepath in zip(images, filepaths):  # loop through the variations\n        with open(filepath, \"wb\") as image_file:  # open the file\n            image_file.write(image)  # write the image to the file\n\n    return filepaths\n```\n\n\n```python\ndalle_prompt = '''\nFull length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.\nShowcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.\nThis piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.\n'''\n```\n\n\n```python\n# Generate your images\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=dalle_prompt,\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n```\n\n\n```python\nfilepaths = process_dalle_images(generation_response, \"generation\", base_image_dir)\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 53d27639-1aed-4bda-a984-4d46365f1933
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"53d27639-1aed-4bda-a984-4d46365f1933\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"502db638-9743-42da-b129-274f9d9d7c53\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4e86934f578d8177e2a0ac14fab1c95638fa3de55e169cc798a89faa5d3e9eff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f7f0568eb9192a7ca1476f462fa08aef7b0dfafa43f719ea9763c74f285d8b68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2758, \"end_char_idx\": 5595, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "```python\ndef process_dalle_images(response, filename, image_dir):\n    # save the images\n    urls = [datum.url for datum in response.data]  # extract URLs\n    images = [requests.get(url).content for url in urls]  # download images\n    image_names = [f\"{filename}_{i + 1}.png\" for i in range(len(images))]  # create names\n    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths\n    for image, filepath in zip(images, filepaths):  # loop through the variations\n        with open(filepath, \"wb\") as image_file:  # open the file\n            image_file.write(image)  # write the image to the file\n\n    return filepaths\n```\n\n\n```python\ndalle_prompt = '''\nFull length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.\nShowcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.\nThis piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.\n'''\n```\n\n\n```python\n# Generate your images\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=dalle_prompt,\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n```\n\n\n```python\nfilepaths = process_dalle_images(generation_response, \"generation\", base_image_dir)\n```\n\n\n```python\n# print the new generations\nfor filepath in filepaths:\n    print(filepath)\n    display(Image.open(filepath))\n```\n\n## Generate Mask\n\nNext we'll load up one of our images and generate masks.\n\nFor this demonstration we're picking a UX where we \"click\" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.\n\n\n```python\n# Pick one of your generated images\nchosen_image = \"images/01_generations/generation_2.png\"\n```\n\n\n```python\n# Function to display mask using matplotlib\ndef show_mask(mask, ax):\n    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])\n    h, w = mask.shape[-2:]\n    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)\n    ax.imshow(mask_image)\n\n\n# Function to display where we've \"clicked\"\ndef show_points(coords, labels, ax, marker_size=375):\n    pos_points = coords[labels == 1]\n    neg_points = coords[labels == 0]\n    ax.scatter(\n        pos_points[:, 0],\n        pos_points[:, 1],\n        color=\"green\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n    ax.scatter(\n        neg_points[:, 0],\n        neg_points[:, 1],\n        color=\"red\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"53d27639-1aed-4bda-a984-4d46365f1933\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b110e4ff66400a98b381fd4a4f322e9749517964b58f3727f8dd72bc1b7a2b23\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bb859304-a79c-4087-a156-ba037633dc0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2c3c5989065f48bf1f980e73e021ea0346f149315b2fcb667411b0f08938bfbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4087, \"end_char_idx\": 7011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "```python\n# print the new generations\nfor filepath in filepaths:\n    print(filepath)\n    display(Image.open(filepath))\n```\n\n## Generate Mask\n\nNext we'll load up one of our images and generate masks.\n\nFor this demonstration we're picking a UX where we \"click\" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.\n\n\n```python\n# Pick one of your generated images\nchosen_image = \"images/01_generations/generation_2.png\"\n```\n\n\n```python\n# Function to display mask using matplotlib\ndef show_mask(mask, ax):\n    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])\n    h, w = mask.shape[-2:]\n    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)\n    ax.imshow(mask_image)\n\n\n# Function to display where we've \"clicked\"\ndef show_points(coords, labels, ax, marker_size=375):\n    pos_points = coords[labels == 1]\n    neg_points = coords[labels == 0]\n    ax.scatter(\n        pos_points[:, 0],\n        pos_points[:, 1],\n        color=\"green\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n    ax.scatter(\n        neg_points[:, 0],\n        neg_points[:, 1],\n        color=\"red\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n\n```\n\n\n```python\n# Load chosen image using opencv\nimage = cv2.imread(chosen_image)\nimage = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n# Display our chosen image\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_14_0.png)\n    \n\n\n\n```python\n# Set the pixel coordinates for our \"click\" to assign masks\ninput_point = np.array([[525, 325]])\ninput_label = np.array([1])\n\n# Display the point we've clicked on\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nshow_points(input_point, input_label, plt.gca())\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_15_0.png)\n    \n\n\n\n```python\n# Initiate predictor with Segment Anything model\npredictor = SamPredictor(sam)\npredictor.set_image(image)\n\n# Use the predictor to gather masks for the point we clicked\nmasks, scores, logits = predictor.predict(\n    point_coords=input_point,\n    point_labels=input_label,\n    multimask_output=True,\n)\n\n# Check the shape - should be three masks of the same dimensions as our image\nmasks.shape\n```\n\n\n\n\n    (3, 1024, 1024)\n\n\n\n\n```python\n# Display the possible masks we can select along with their confidence\nfor i, (mask, score) in enumerate(zip(masks, scores)):\n    plt.figure(figsize=(10, 10))\n    plt.imshow(image)\n    show_mask(mask, plt.gca())\n    show_points(input_point, input_label, plt.gca())\n    plt.title(f\"Mask {i+1}, Score: {score:.3f}\", fontsize=18)\n    plt.axis(\"off\")\n    plt.show()\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bb859304-a79c-4087-a156-ba037633dc0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bb859304-a79c-4087-a156-ba037633dc0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"be3086ce650e376b704f40498b5eb54d83401987d9e2089deb0beef613d83aa4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e95226ce-9e7c-432d-997e-cbb00cfc7b2b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cdae518d1ba8cdab98b04dbdfd1b6bd19ac151af4ca5491eb795369b603c069a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5598, \"end_char_idx\": 8235, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "```python\n# Load chosen image using opencv\nimage = cv2.imread(chosen_image)\nimage = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n# Display our chosen image\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_14_0.png)\n    \n\n\n\n```python\n# Set the pixel coordinates for our \"click\" to assign masks\ninput_point = np.array([[525, 325]])\ninput_label = np.array([1])\n\n# Display the point we've clicked on\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nshow_points(input_point, input_label, plt.gca())\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_15_0.png)\n    \n\n\n\n```python\n# Initiate predictor with Segment Anything model\npredictor = SamPredictor(sam)\npredictor.set_image(image)\n\n# Use the predictor to gather masks for the point we clicked\nmasks, scores, logits = predictor.predict(\n    point_coords=input_point,\n    point_labels=input_label,\n    multimask_output=True,\n)\n\n# Check the shape - should be three masks of the same dimensions as our image\nmasks.shape\n```\n\n\n\n\n    (3, 1024, 1024)\n\n\n\n\n```python\n# Display the possible masks we can select along with their confidence\nfor i, (mask, score) in enumerate(zip(masks, scores)):\n    plt.figure(figsize=(10, 10))\n    plt.imshow(image)\n    show_mask(mask, plt.gca())\n    show_points(input_point, input_label, plt.gca())\n    plt.title(f\"Mask {i+1}, Score: {score:.3f}\", fontsize=18)\n    plt.axis(\"off\")\n    plt.show()\n```\n\n\n    \n![png](output_17_0.png)\n    \n\n\n\n    \n![png](output_17_1.png)\n    \n\n\n\n    \n![png](output_17_2.png)\n    \n\n\n\n```python\n# Choose which mask you'd like to use\nchosen_mask = masks[1]\n\n# We'll now reverse the mask so that it is clear and everything else is white\nchosen_mask = chosen_mask.astype(\"uint8\")\nchosen_mask[chosen_mask != 0] = 255\nchosen_mask[chosen_mask == 0] = 1\nchosen_mask[chosen_mask == 255] = 0\nchosen_mask[chosen_mask == 1] = 255\n\n```\n\n\n```python\n# create a base blank mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# Convert mask back to pixels to add our mask replacing the third dimension\npix = np.array(mask)\npix[:, :, 3] = chosen_mask\n\n# Convert pixels back to an RGBA image and display\nnew_mask = Image.fromarray(pix, \"RGBA\")\nnew_mask\n```\n\n\n\n\n    \n![png](output_19_0.png)\n    \n\n\n\n\n```python\n# We'll save this mask for re-use for our edit\nnew_mask.save(os.path.join(mask_dir, \"new_mask.png\"))\n```\n\n## Create new image\n\nNow we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e95226ce-9e7c-432d-997e-cbb00cfc7b2b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e95226ce-9e7c-432d-997e-cbb00cfc7b2b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"5cb07a00-3915-49b2-a67f-75cefa39c831\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bb859304-a79c-4087-a156-ba037633dc0d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9a231d1a902374e24bfcc4f379204c06bab6ad922fc50e08399ad03bff73be90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7019, \"end_char_idx\": 9362, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "ref_doc_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "chroma:document": "![png](output_17_0.png)\n    \n\n\n\n    \n![png](output_17_1.png)\n    \n\n\n\n    \n![png](output_17_2.png)\n    \n\n\n\n```python\n# Choose which mask you'd like to use\nchosen_mask = masks[1]\n\n# We'll now reverse the mask so that it is clear and everything else is white\nchosen_mask = chosen_mask.astype(\"uint8\")\nchosen_mask[chosen_mask != 0] = 255\nchosen_mask[chosen_mask == 0] = 1\nchosen_mask[chosen_mask == 255] = 0\nchosen_mask[chosen_mask == 1] = 255\n\n```\n\n\n```python\n# create a base blank mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# Convert mask back to pixels to add our mask replacing the third dimension\npix = np.array(mask)\npix[:, :, 3] = chosen_mask\n\n# Convert pixels back to an RGBA image and display\nnew_mask = Image.fromarray(pix, \"RGBA\")\nnew_mask\n```\n\n\n\n\n    \n![png](output_19_0.png)\n    \n\n\n\n\n```python\n# We'll save this mask for re-use for our edit\nnew_mask.save(os.path.join(mask_dir, \"new_mask.png\"))\n```\n\n## Create new image\n\nNow we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)\n\n\n\n```python\n# edit an image\nedit_response = client.images.edit(\n    image=open(chosen_image, \"rb\"),  # from the generation section\n    mask=open(os.path.join(mask_dir, \"new_mask.png\"), \"rb\"),  # from right above\n    prompt=\"Brilliant leather Lederhosen with a formal look, detailed, intricate, photorealistic\",  # provide a prompt to fill the space\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\nedit_filepaths = process_dalle_images(edit_response, \"edits\", edit_image_dir)\n\n```\n\n\n```python\n# Display your beautiful creations!\n%matplotlib inline\n\n# figure size in inches optional\nrcParams[\"figure.figsize\"] = 11 ,8\n\n# read images\nimg_A = mpimg.imread(edit_filepaths[0])\nimg_B = mpimg.imread(edit_filepaths[1])\nimg_C = mpimg.imread(edit_filepaths[2])\n\n# display images\nfig, ax = plt.subplots(1,3)\n[a.axis(\"off\") for a in ax]\nax[0].imshow(img_A)\nax[1].imshow(img_B)\nax[2].imshow(img_C)\n```\n\n\n\n\n    <matplotlib.image.AxesImage at 0x791b1f4c58a0>\n\n\n\n\n    \n![png](output_23_1.png)\n    \n\n\nBeautiful!\n\nNow you too can easily create dynamic masks to extend your images - enjoy the APIs, and please share what you build!"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a6300a41-e6e1-4f44-b1d8-019d1634f0dd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a6300a41-e6e1-4f44-b1d8-019d1634f0dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b76357f-8c0b-4717-a49c-deacfc601807\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"95c5cdd8a888a4a50454d7672f9c198c1edabd1ce5a706351e5f15ba5e51e17c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2450, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "# How to evaluate a summarization task\n\nIn this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like [ROUGE](https://aclanthology.org/W04-1013/) and [BERTScore](https://arxiv.org/abs/1904.09675), in addition to showcasing a more novel approach using LLMs as evaluators.\n\nEvaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as `ROUGE` and `BERTScore` and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks ([Liu et al., 2023](https://arxiv.org/pdf/2303.16634.pdf)). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive.\n\nIn addition to these traditional metrics, we showcase a method ([G-Eval](https://arxiv.org/pdf/2303.16634.pdf)) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use `gpt-4` to score candidate outputs. `gpt-4` has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.\n\n\n## Setup\n\n\n\n```python\n# Installing necessary packages for the evaluation\n# rouge: For evaluating with ROUGE metric\n# bert_score: For evaluating with BERTScore\n# openai: To interact with OpenAI's API\n!pip install rouge --quiet\n!pip install bert_score --quiet\n!pip install openai --quiet\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport pandas as pd\n\n# Python Implementation of the ROUGE Metric\nfrom rouge import Rouge\n\n# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.\nfrom bert_score import BERTScorer\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n    <IPython.core.display.Javascript object>"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7b76357f-8c0b-4717-a49c-deacfc601807
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7b76357f-8c0b-4717-a49c-deacfc601807\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6300a41-e6e1-4f44-b1d8-019d1634f0dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8578004bb41ff667a1b8b98372d06a46f8ea23c2379d6efd592885bc255f93f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ae4f21a9-b32f-4632-a6fe-58e7d0598087\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8f266d2cac970d57cd8bde60d8d84155be9ea038460f0370ee9be4541f6362e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1674, \"end_char_idx\": 6892, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "## Setup\n\n\n\n```python\n# Installing necessary packages for the evaluation\n# rouge: For evaluating with ROUGE metric\n# bert_score: For evaluating with BERTScore\n# openai: To interact with OpenAI's API\n!pip install rouge --quiet\n!pip install bert_score --quiet\n!pip install openai --quiet\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport pandas as pd\n\n# Python Implementation of the ROUGE Metric\nfrom rouge import Rouge\n\n# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.\nfrom bert_score import BERTScorer\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n    <IPython.core.display.Javascript object>\n\n\n## Example task\n\nFor the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like `ROUGE` and `BERTScore` require.\n\nExcerpt (`excerpt`):\n\n> OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.\n\nSummaries:\n\n| Reference Summary /`ref_summary` (human generated)                                                                                                                                                                                                                                                                                                                         | Eval Summary 1 / `eval_summary_1` (system generated)                                                                                                                                                                                                                                                                                                                               | Eval Summary 2 / `eval_summary_2` (system generated)                                                                                                                                                                                                                                                   |\n| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges. | OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good. | OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff. |\n\nTake a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ae4f21a9-b32f-4632-a6fe-58e7d0598087
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ae4f21a9-b32f-4632-a6fe-58e7d0598087\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b76357f-8c0b-4717-a49c-deacfc601807\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bcd5efef4d3e0374c3038373bc10e4bf182c829c53e5f774372c57bde7f1e8c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e811d8fe-58c1-46e7-bb75-8499e6544374\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4521af41da3037c5cbad4fccbc8f93dac9bc5779bbf896d17fe26f9b4101f53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6896, \"end_char_idx\": 10220, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "```python\nexcerpt = \"OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.\"\nref_summary = \"OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges.\"\neval_summary_1 = \"OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good.\"\neval_summary_2 = \"OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff.\"\n```\n\n\n    <IPython.core.display.Javascript object>\n\n\n## Evaluating using ROUGE\n\n[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.\n\n\n\n```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e811d8fe-58c1-46e7-bb75-8499e6544374
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e811d8fe-58c1-46e7-bb75-8499e6544374\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae4f21a9-b32f-4632-a6fe-58e7d0598087\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"efe0754d2c2fb1fa89affb8e8a9e725591765f7528f95a624573f5bbf172737e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c79b6a6c-b173-41f9-b88e-9ad345c6835f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81476445bc57001c5fb5142b26eee5186b5a76ce47defb6e9d76e4d65e7bb780\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8924, \"end_char_idx\": 10535, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "<IPython.core.display.Javascript object>\n\n\n## Evaluating using ROUGE\n\n[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.\n\n\n\n```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)\n\n\ndef highlight_max(s):\n    is_max = s == s.max()\n    return [\n        \"background-color: lightgreen\" if v else \"background-color: white\"\n        for v in is_max\n    ]\n\n\nrouge_scores_out = (\n    pd.DataFrame(rouge_scores_out)\n    .set_index(\"Metric\")\n    .style.apply(highlight_max, axis=1)\n)\n\nrouge_scores_out\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c79b6a6c-b173-41f9-b88e-9ad345c6835f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c79b6a6c-b173-41f9-b88e-9ad345c6835f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e811d8fe-58c1-46e7-bb75-8499e6544374\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1874c5528d117efd82bc0214e88db51f39ea647bf1f1f20d7caa929dca48ca05\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c1addd0a-8c54-4009-a6a9-040ff886f2c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eee3558fa20b8944ebcd928b79ba588daac76dc4eed49cb5bec8a0f8df83185d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9465, \"end_char_idx\": 11966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)\n\n\ndef highlight_max(s):\n    is_max = s == s.max()\n    return [\n        \"background-color: lightgreen\" if v else \"background-color: white\"\n        for v in is_max\n    ]\n\n\nrouge_scores_out = (\n    pd.DataFrame(rouge_scores_out)\n    .set_index(\"Metric\")\n    .style.apply(highlight_max, axis=1)\n)\n\nrouge_scores_out\n```\n\n\n\n\n<style type=\"text/css\">\n#T_7e6ac_row0_col0, #T_7e6ac_row1_col1, #T_7e6ac_row2_col0 {\n  background-color: white;\n}\n#T_7e6ac_row0_col1, #T_7e6ac_row1_col0, #T_7e6ac_row2_col1 {\n  background-color: lightgreen;\n}\n</style>\n<table id=\"T_7e6ac\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_7e6ac_level0_col0\" class=\"col_heading level0 col0\" >Summary 1</th>\n      <th id=\"T_7e6ac_level0_col1\" class=\"col_heading level0 col1\" >Summary 2</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level0\" >Metric</th>\n      <th class=\"blank col0\" >&nbsp;</th>\n      <th class=\"blank col1\" >&nbsp;</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_7e6ac_level0_row0\" class=\"row_heading level0 row0\" >rouge-1 (F-Score)</th>\n      <td id=\"T_7e6ac_row0_col0\" class=\"data row0 col0\" >0.488889</td>\n      <td id=\"T_7e6ac_row0_col1\" class=\"data row0 col1\" >0.511628</td>\n    </tr>\n    <tr>\n      <th id=\"T_7e6ac_level0_row1\" class=\"row_heading level0 row1\" >rouge-2 (F-Score)</th>\n      <td id=\"T_7e6ac_row1_col0\" class=\"data row1 col0\" >0.230769</td>\n      <td id=\"T_7e6ac_row1_col1\" class=\"data row1 col1\" >0.163265</td>\n    </tr>\n    <tr>\n      <th id=\"T_7e6ac_level0_row2\" class=\"row_heading level0 row2\" >rouge-l (F-Score)</th>\n      <td id=\"T_7e6ac_row2_col0\" class=\"data row2 col0\" >0.488889</td>\n      <td id=\"T_7e6ac_row2_col1\" class=\"data row2 col1\" >0.511628</td>\n    </tr>\n  </tbody>\n</table>"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c1addd0a-8c54-4009-a6a9-040ff886f2c2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c1addd0a-8c54-4009-a6a9-040ff886f2c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c79b6a6c-b173-41f9-b88e-9ad345c6835f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d1e6ae6332f47d3c758e8b71f696c7631e657cab5f88b07bfcc91abd14b1ab7f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d01e0636fc392f1e46e62a08904eaaf217b146c4946a8d5c24e556b92a550980\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11976, \"end_char_idx\": 15206, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "<IPython.core.display.Javascript object>\n\n\nThe table shows the `ROUGE` scores for evaluating two different summaries against a reference text. In the case of `rouge-1`, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for `rouge-l`, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text. Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher `ROUGE` scores.\n\nWhile `ROUGE` and similar metrics, such as [BLEU](https://aclanthology.org/P02-1040.pdf) and [METEOR](https://www.cs.cmu.edu/~alavie/METEOR/), offer quantitative measures, they often fail to capture the true essence of a well-generated summary. They also correlate worse with human scores. Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like `ROUGE` may inadvertently penalize these models. This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.\n\n\n## Evaluating using BERTScore\n\nROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where [BERTScore](https://arxiv.org/abs/1904.09675) comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, `BERTScore` captures semantic similarities that might be missed by traditional n-gram based metrics.\n\n\n\n```python\n# Instantiate the BERTScorer object for English language\nscorer = BERTScorer(lang=\"en\")\n\n# Calculate BERTScore for the summary 1 against the excerpt\n# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively\nP1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])\n\n# Calculate BERTScore for summary 2 against the excerpt\n# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively\nP2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])\n\nprint(\"Summary 1 F1 Score:\", F1_1.tolist()[0])\nprint(\"Summary 2 F1 Score:\", F2_2.tolist()[0])\n```\n\n    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']\n    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n    Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c1addd0a-8c54-4009-a6a9-040ff886f2c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1921a14c7b2200582b49bfb9715a0dd8d7139b8372a29ebf1a30d07ea523a431\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ac1580bc-a4ec-430d-93be-2c363bcd3536\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1439f8bb4fc7a2a79da913830b7604d9dd81f85ace5d5ffb1ac5bb2d0d6db099\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13777, \"end_char_idx\": 15737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "```python\n# Instantiate the BERTScorer object for English language\nscorer = BERTScorer(lang=\"en\")\n\n# Calculate BERTScore for the summary 1 against the excerpt\n# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively\nP1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])\n\n# Calculate BERTScore for summary 2 against the excerpt\n# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively\nP2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])\n\nprint(\"Summary 1 F1 Score:\", F1_1.tolist()[0])\nprint(\"Summary 2 F1 Score:\", F2_2.tolist()[0])\n```\n\n    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']\n    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n    Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>\n\n\nThe close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ac1580bc-a4ec-430d-93be-2c363bcd3536
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ac1580bc-a4ec-430d-93be-2c363bcd3536\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9ff7f98557cccf735f7e9dd40ae3397da730c142d8676afed535e5c6e370d010\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7f809bd-cb1a-4cbe-9ae3-f051066542ba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c7abc59b4b1407c97964e334311db93595655397b1fa1b4054e0aa57e13bb992\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15077, \"end_char_idx\": 17368, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>\n\n\nThe close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.\n\n\n## Evaluating using GPT-4\n\nHere we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. \n\nHere's an overview of this method:\n\n1. We define four distinct criteria:\n    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.\n    2. **Coherence**: Assesses the logical flow and organization of the summary.\n    3. **Consistency**: Checks if the summary aligns with the facts in the source document.\n    4. **Fluency**: Rates the grammar and readability of the summary.\n2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. \n3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.\n\nIn this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b7f809bd-cb1a-4cbe-9ae3-f051066542ba
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b7f809bd-cb1a-4cbe-9ae3-f051066542ba\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ac1580bc-a4ec-430d-93be-2c363bcd3536\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0264a51a7f0892b2fcda8c49649bb0b9ecb60fa210ec8d368479c340f847b5f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de7dfc0d-3e15-4725-87b6-6106e46ecd42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23a177c88aa82909084813256b18dd5831fa17f2ed7924b26ca8c4ea6db46ade\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15740, \"end_char_idx\": 20652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "## Evaluating using GPT-4\n\nHere we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. \n\nHere's an overview of this method:\n\n1. We define four distinct criteria:\n    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.\n    2. **Coherence**: Assesses the logical flow and organization of the summary.\n    3. **Consistency**: Checks if the summary aligns with the facts in the source document.\n    4. **Fluency**: Rates the grammar and readability of the summary.\n2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. \n3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.\n\nIn this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.\n\n\n```python\n# Evaluation prompt template based on G-Eval\nEVALUATION_PROMPT_TEMPLATE = \"\"\"\nYou will be given one summary written for an article. Your task is to rate the summary on one metric.\nPlease make sure you read and understand these instructions very carefully. \nPlease keep this document open while reviewing, and refer to it as needed.\n\nEvaluation Criteria:\n\n{criteria}\n\nEvaluation Steps:\n\n{steps}\n\nExample:\n\nSource Text:\n\n{document}\n\nSummary:\n\n{summary}\n\nEvaluation Form (scores ONLY):\n\n- {metric_name}\n\"\"\"\n\n# Metric 1: Relevance\n\nRELEVANCY_SCORE_CRITERIA = \"\"\"\nRelevance(1-5) - selection of important content from the source. \\\nThe summary should include only important information from the source document. \\\nAnnotators were instructed to penalize summaries which contained redundancies and excess information.\n\"\"\"\n\nRELEVANCY_SCORE_STEPS = \"\"\"\n1. Read the summary and the source document carefully.\n2. Compare the summary to the source document and identify the main points of the article.\n3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.\n4. Assign a relevance score from 1 to 5.\n\"\"\"\n\n# Metric 2: Coherence\n\nCOHERENCE_SCORE_CRITERIA = \"\"\"\nCoherence(1-5) - the collective quality of all sentences. \\\nWe align this dimension with the DUC quality question of structure and coherence \\\nwhereby \"the summary should be well-structured and well-organized. \\\nThe summary should not just be a heap of related information, but should build from sentence to a\\\ncoherent body of information about a topic.\"\n\"\"\"\n\nCOHERENCE_SCORE_STEPS = \"\"\"\n1. Read the article carefully and identify the main topic and key points.\n2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,\nand if it presents them in a clear and logical order.\n3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.\n\"\"\"\n\n# Metric 3: Consistency\n\nCONSISTENCY_SCORE_CRITERIA = \"\"\"\nConsistency(1-5) - the factual alignment between the summary and the summarized source. \\\nA factually consistent summary contains only statements that are entailed by the source document. \\\nAnnotators were also asked to penalize summaries that contained hallucinated facts.\n\"\"\"\n\nCONSISTENCY_SCORE_STEPS = \"\"\"\n1. Read the article carefully and identify the main facts and details it presents.\n2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.\n3. Assign a score for consistency based on the Evaluation Criteria.\n\"\"\"\n\n# Metric 4: Fluency\n\nFLUENCY_SCORE_CRITERIA = \"\"\"\nFluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.\n1: Poor. The summary has many errors that make it hard to understand or sound unnatural.\n2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.\n3: Good. The summary has few or no errors and is easy to read and follow.\n\"\"\"\n\nFLUENCY_SCORE_STEPS = \"\"\"\nRead the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.\n\"\"\""}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: de7dfc0d-3e15-4725-87b6-6106e46ecd42
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"de7dfc0d-3e15-4725-87b6-6106e46ecd42\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7f809bd-cb1a-4cbe-9ae3-f051066542ba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cca993148068c170e12e6ef7c89d1af0b4bc69c2e57cae4514119ec691c061df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2b61792e-b10e-41a7-818d-d8baa65d00c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ea6e35c127f1e8581148d8689708c8b5e08709865a2b0e091e648c8e0b35793\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20655, \"end_char_idx\": 22257, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "def get_geval_score(\n    criteria: str, steps: str, document: str, summary: str, metric_name: str\n):\n    prompt = EVALUATION_PROMPT_TEMPLATE.format(\n        criteria=criteria,\n        steps=steps,\n        metric_name=metric_name,\n        document=document,\n        summary=summary,\n    )\n    response = client.chat.completions.create(\n        model=\"gpt-4\",\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n        temperature=0,\n        max_tokens=5,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n    )\n    return response.choices[0].message.content\n\n\nevaluation_metrics = {\n    \"Relevance\": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),\n    \"Coherence\": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),\n    \"Consistency\": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),\n    \"Fluency\": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),\n}\n\nsummaries = {\"Summary 1\": eval_summary_1, \"Summary 2\": eval_summary_2}\n\ndata = {\"Evaluation Type\": [], \"Summary Type\": [], \"Score\": []}\n\nfor eval_type, (criteria, steps) in evaluation_metrics.items():\n    for summ_type, summary in summaries.items():\n        data[\"Evaluation Type\"].append(eval_type)\n        data[\"Summary Type\"].append(summ_type)\n        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)\n        score_num = int(result.strip())\n        data[\"Score\"].append(score_num)\n\npivot_df = pd.DataFrame(data, index=None).pivot(\n    index=\"Evaluation Type\", columns=\"Summary Type\", values=\"Score\"\n)\nstyled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)\ndisplay(styled_pivot_df)\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2b61792e-b10e-41a7-818d-d8baa65d00c7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2b61792e-b10e-41a7-818d-d8baa65d00c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"de7dfc0d-3e15-4725-87b6-6106e46ecd42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7d01dbd435f8cf90e8365309a81d16b57f3165bcb272d3633c1739109db2e15d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3e99123-8fa4-4d23-a1b1-3e744915a2f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88d39cc6f73eb420872c3104aa885fca7521be5c7395c9bcdd5bf0637f81b1e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21243, \"end_char_idx\": 23910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "evaluation_metrics = {\n    \"Relevance\": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),\n    \"Coherence\": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),\n    \"Consistency\": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),\n    \"Fluency\": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),\n}\n\nsummaries = {\"Summary 1\": eval_summary_1, \"Summary 2\": eval_summary_2}\n\ndata = {\"Evaluation Type\": [], \"Summary Type\": [], \"Score\": []}\n\nfor eval_type, (criteria, steps) in evaluation_metrics.items():\n    for summ_type, summary in summaries.items():\n        data[\"Evaluation Type\"].append(eval_type)\n        data[\"Summary Type\"].append(summ_type)\n        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)\n        score_num = int(result.strip())\n        data[\"Score\"].append(score_num)\n\npivot_df = pd.DataFrame(data, index=None).pivot(\n    index=\"Evaluation Type\", columns=\"Summary Type\", values=\"Score\"\n)\nstyled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)\ndisplay(styled_pivot_df)\n```\n\n\n<style type=\"text/css\">\n#T_94fab_row0_col0, #T_94fab_row1_col0, #T_94fab_row1_col1, #T_94fab_row2_col0, #T_94fab_row3_col0 {\n  background-color: lightgreen;\n}\n#T_94fab_row0_col1, #T_94fab_row2_col1, #T_94fab_row3_col1 {\n  background-color: white;\n}\n</style>\n<table id=\"T_94fab\">\n  <thead>\n    <tr>\n      <th class=\"index_name level0\" >Summary Type</th>\n      <th id=\"T_94fab_level0_col0\" class=\"col_heading level0 col0\" >Summary 1</th>\n      <th id=\"T_94fab_level0_col1\" class=\"col_heading level0 col1\" >Summary 2</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level0\" >Evaluation Type</th>\n      <th class=\"blank col0\" >&nbsp;</th>\n      <th class=\"blank col1\" >&nbsp;</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_94fab_level0_row0\" class=\"row_heading level0 row0\" >Coherence</th>\n      <td id=\"T_94fab_row0_col0\" class=\"data row0 col0\" >5</td>\n      <td id=\"T_94fab_row0_col1\" class=\"data row0 col1\" >3</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row1\" class=\"row_heading level0 row1\" >Consistency</th>\n      <td id=\"T_94fab_row1_col0\" class=\"data row1 col0\" >5</td>\n      <td id=\"T_94fab_row1_col1\" class=\"data row1 col1\" >5</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row2\" class=\"row_heading level0 row2\" >Fluency</th>\n      <td id=\"T_94fab_row2_col0\" class=\"data row2 col0\" >3</td>\n      <td id=\"T_94fab_row2_col1\" class=\"data row2 col1\" >2</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row3\" class=\"row_heading level0 row3\" >Relevance</th>\n      <td id=\"T_94fab_row3_col0\" class=\"data row3 col0\" >5</td>\n      <td id=\"T_94fab_row3_col1\" class=\"data row3 col1\" >4</td>\n    </tr>\n  </tbody>\n</table>"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d3e99123-8fa4-4d23-a1b1-3e744915a2f0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d3e99123-8fa4-4d23-a1b1-3e744915a2f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dab62be3-40f5-4f6c-8c90-010816f48e4d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2b61792e-b10e-41a7-818d-d8baa65d00c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"23452416588d307b2144d8a52eef91571b5a5aaf3dc9488a1a87cf5043dfba1c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23919, \"end_char_idx\": 26471, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "ref_doc_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "chroma:document": "<IPython.core.display.Javascript object>\n\n\n\nOverall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency). Both summaries are found to be consistent with each other. The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.\n\n### Limitations\n\nNote that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by `gpt-4`'s context window.\n\n## Conclusion\n\nEvaluating abstractive summarization remains an open area for further improvement. Traditional metrics like `ROUGE`, `BLEU`, and `BERTScore` provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.\n\n## References\n\n- [G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment](https://arxiv.org/pdf/2303.16634.pdf) - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. Published May, 2023.\n- [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Published online February, 2020.\n- [ROUGE: A Package for Automatic Evaluation of Summaries](https://aclanthology.org/W04-1013/) - Lin CY. Published July, 2004.\n- [SummEval: Re-evaluating Summarization Evaluation](https://aclanthology.org/2021.tacl-1.24) - Fabbri et al. Published April, 2021."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bc340776-f54a-47e6-960e-542283aa0dc7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bc340776-f54a-47e6-960e-542283aa0dc7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23fc111e-8df6-433c-9a2e-2b76c9871f25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab4c3631d8abc55b383c576a8b460c6c41062f442df7d0e09a6ead8c1240672b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3413, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "# Getting Started with OpenAI Evals\n\nThe [OpenAI Evals](https://github.com/openai/evals/tree/main) framework consists of\n1. A framework to evaluate an LLM (large language model) or a system built on top of an LLM.\n2. An open-source registry of challenging evals\n\nThis notebook will cover:\n* Introduction to Evaluation and the [OpenAI Evals](https://github.com/openai/evals/tree/main) library\n* Building an Eval\n* Running an Eval\n\n#### What are evaluations/ `evals`?\n\nEvaluation is the process of validating and testing the outputs that your LLM applications are producing. Having strong evaluations (\"evals\") will mean a more stable, reliable application that is resilient to code and model changes. An eval is a task used to measure the quality of the output of an LLM or LLM system. Given an input prompt, an output is generated. We evaluate this output with a set of ideal answers and find the quality of the LLM system.\n\n#### Importance of Evaluations\n\nIf you are building with foundational models like `GPT-4`, creating high quality evals is one of the most impactful things you can do. Developing AI solutions involves an iterative design process. [Without evals, it can be very difficult and time intensive to understand](https://youtu.be/XGJNo8TpuVA?feature=shared&t=1089) how different model versions and prompts might affect your use case.\n\nWith OpenAI\u2019s [continuous model upgrades](https://platform.openai.com/docs/models/continuous-model-upgrades), evals allow you to efficiently test model performance for your use cases in a standardized way. Developing a suite of evals customized to your objectives will help you quickly and effectively understand how new models may perform for your use cases. You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.\n\n#### Types of evals\n\nThere are two main ways we can evaluate/grade completions: writing some validation logic in code\nor using the model itself to inspect the answer. We\u2019ll introduce each with some examples.\n\n**Writing logic for answer checking**\n\nThe simplest and most common type of eval has an input and an ideal response or answer. For example,\nwe can have an eval sample where the input is \"What year was Obama elected president for the first\ntime?\" and the ideal answer is \"2008\". We feed the input to a model and get the completion. If the model\nsays \"2008\", it is then graded as correct. We can write a string match to check if the completion includes the phrase \"2008\". If it does, we consider it correct.\n\nConsider another eval where the input is to generate valid JSON: We can write some code that\nattempts to parse the completion as JSON and then considers the completion correct if it is\nparsable.\n\n**Model grading: A two stage process where the model first answers the question, then we ask a\nmodel to look at the response to check if it\u2019s correct.**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 23fc111e-8df6-433c-9a2e-2b76c9871f25
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"23fc111e-8df6-433c-9a2e-2b76c9871f25\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bc340776-f54a-47e6-960e-542283aa0dc7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"38029a5b18c45432a1b640575a06a0261f607c874da72bab845523d1d2ab7152\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4c967fc9-fd7d-47e7-888a-a35ed1bd8f30\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0fc158e077f087629de75301c3a8ad9ef30edf09d88ff0e6a6816311e60c4f1a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1711, \"end_char_idx\": 4725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.\n\n#### Types of evals\n\nThere are two main ways we can evaluate/grade completions: writing some validation logic in code\nor using the model itself to inspect the answer. We\u2019ll introduce each with some examples.\n\n**Writing logic for answer checking**\n\nThe simplest and most common type of eval has an input and an ideal response or answer. For example,\nwe can have an eval sample where the input is \"What year was Obama elected president for the first\ntime?\" and the ideal answer is \"2008\". We feed the input to a model and get the completion. If the model\nsays \"2008\", it is then graded as correct. We can write a string match to check if the completion includes the phrase \"2008\". If it does, we consider it correct.\n\nConsider another eval where the input is to generate valid JSON: We can write some code that\nattempts to parse the completion as JSON and then considers the completion correct if it is\nparsable.\n\n**Model grading: A two stage process where the model first answers the question, then we ask a\nmodel to look at the response to check if it\u2019s correct.**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment. Model grading will have an error rate, so it is important to validate\nthe performance with human evaluation before running the evals at scale. For best results, it makes\nsense to use a different model to do grading from the one that did the completion, like using `GPT-4` to\ngrade `GPT-3.5` answers.\n\n#### OpenAI Eval Templates\n\nIn using evals, we have discovered several \"templates\" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:\n\n* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.\n\n* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4c967fc9-fd7d-47e7-888a-a35ed1bd8f30
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4c967fc9-fd7d-47e7-888a-a35ed1bd8f30\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23fc111e-8df6-433c-9a2e-2b76c9871f25\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cb89f44b483d26f496649d703364a80a49127560da5172618a243c0bb6ca2e30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d88cec30-e3f7-48d6-97dd-466844bd54d6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef15426fa16a5af3c64c0c7b38afeb6eb2d83277228ce31476ef3c04a0a2e8e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2892, \"end_char_idx\": 6042, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment. Model grading will have an error rate, so it is important to validate\nthe performance with human evaluation before running the evals at scale. For best results, it makes\nsense to use a different model to do grading from the one that did the completion, like using `GPT-4` to\ngrade `GPT-3.5` answers.\n\n#### OpenAI Eval Templates\n\nIn using evals, we have discovered several \"templates\" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:\n\n* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.\n\n* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.\n\n\n### Getting Setup\n\nFirst, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). \n\nTo run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. \n\nPlease be aware of the costs associated with using the API when running evals.\n\n\n```python\nfrom openai import OpenAI\nimport pandas as pd\n\nclient = OpenAI()\n```\n\n## Building an evaluation for OpenAI Evals framework\n\nAt its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need\n\n1. The test dataset in the `jsonl` format.\n2. The eval template to be used\n\n### Creating the eval dataset\nLets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing\n\nFirst we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d88cec30-e3f7-48d6-97dd-466844bd54d6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d88cec30-e3f7-48d6-97dd-466844bd54d6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4c967fc9-fd7d-47e7-888a-a35ed1bd8f30\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"574d841494e4d3462013d80195796b1eb123d1177ea85cc8d7c5e7d0de4e56b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12ea236f-05b4-4e2f-8468-dc59603f7a14\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"098fa12002a403201c2dc3ce999656ad9eb8764397035ddb0aa8c39930dd9500\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4365, \"end_char_idx\": 7321, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.\n\n\n### Getting Setup\n\nFirst, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). \n\nTo run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. \n\nPlease be aware of the costs associated with using the API when running evals.\n\n\n```python\nfrom openai import OpenAI\nimport pandas as pd\n\nclient = OpenAI()\n```\n\n## Building an evaluation for OpenAI Evals framework\n\nAt its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need\n\n1. The test dataset in the `jsonl` format.\n2. The eval template to be used\n\n### Creating the eval dataset\nLets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing\n\nFirst we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n```\n\nFor this prompt, we can ask a specific question:\n```\n\"Q: how many car makers are their in germany?\"\n```\n\nAnd we have an expected answer:\n```\n\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"\n```\n\nThe dataset needs to be in the following format:\n```\n\"input\": [{\"role\": \"system\", \"content\": \"<input prompt>\"}, {\"role\": \"user\", \"content\": <user input>}, \"ideal\": \"correct answer\"]\n```\n\nPutting it all together, we get:\n```\n{\"input\": [{\"role\": \"system\", \"content\": \"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 12ea236f-05b4-4e2f-8468-dc59603f7a14
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"12ea236f-05b4-4e2f-8468-dc59603f7a14\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d88cec30-e3f7-48d6-97dd-466844bd54d6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e19ffa91b3260dd2e52de284c303fb26d26e6a0dd2ff0c0a3b46a2bfded16f56\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cbbed732-83f1-40fd-8167-955b684f896e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7179d32f86a295a13789a06d4f3696e44fedf34b12da732b4b2e2b6d6a01d54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5873, \"end_char_idx\": 8791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n```\n\nFor this prompt, we can ask a specific question:\n```\n\"Q: how many car makers are their in germany?\"\n```\n\nAnd we have an expected answer:\n```\n\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"\n```\n\nThe dataset needs to be in the following format:\n```\n\"input\": [{\"role\": \"system\", \"content\": \"<input prompt>\"}, {\"role\": \"user\", \"content\": <user input>}, \"ideal\": \"correct answer\"]\n```\n\nPutting it all together, we get:\n```\n{\"input\": [{\"role\": \"system\", \"content\": \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\\n\"}, {\"role\": \"system\", \"content\": \"Q: how many car makers are their in germany\"}, \"ideal\": [\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"]}\n```\n\n\nOne way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data\n\n\n```python\n## Use GPT-4 to generate synthetic data\n# Define the system prompt and user input (these should be filled as per the specific use case)\nsystem_prompt = \"\"\"You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.\n    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cbbed732-83f1-40fd-8167-955b684f896e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cbbed732-83f1-40fd-8167-955b684f896e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12ea236f-05b4-4e2f-8468-dc59603f7a14\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8356451ce944a77314e2b0abb1aa8dbd35df75048f2e97a5ad8ce240eba918e0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b8d4de4-86d6-478c-9e69-84829cc22308\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"944dd52fafe091ffa8cacc3ed42efca11af9cbbaae0e63d1dd6f8567fc92b130\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7322, \"end_char_idx\": 9957, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\\n\"}, {\"role\": \"system\", \"content\": \"Q: how many car makers are their in germany\"}, \"ideal\": [\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"]}\n```\n\n\nOne way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data\n\n\n```python\n## Use GPT-4 to generate synthetic data\n# Define the system prompt and user input (these should be filled as per the specific use case)\nsystem_prompt = \"\"\"You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.\n    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.\n\n    Example 1\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many visitors have visited the museum with the most staff?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6b8d4de4-86d6-478c-9e69-84829cc22308
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6b8d4de4-86d6-478c-9e69-84829cc22308\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cbbed732-83f1-40fd-8167-955b684f896e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"eeec75448ce3ead6304e2bb70d57889ac18db5235cfedca50b487978f2195d6a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b1b4b880-25ec-4307-ba60-f1304aed71e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b20c3a7abd7d0dd5b2d836298bfb85360eeb3498b6d85f8a6af5dc2f9a7883e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8604, \"end_char_idx\": 10452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.\n\n    Example 1\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many visitors have visited the museum with the most staff?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?\n    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 \n    ```````````\n\n    Example 3\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many tickets of customer id 5?"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b1b4b880-25ec-4307-ba60-f1304aed71e0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b1b4b880-25ec-4307-ba60-f1304aed71e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6b8d4de4-86d6-478c-9e69-84829cc22308\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5a4a99c1dfca9ed8ad0352124a203601e14d8b94ec13612adde14aa4d4eee072\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ffee6f55-2362-4906-aae9-a7e497df2526\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"36b266f7fb045846f1558880bb533a39b93c3c4c5d4c17b4d9c8482a210e7bf2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9347, \"end_char_idx\": 12032, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?\n    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 \n    ```````````\n\n    Example 3\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many tickets of customer id 5?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN VISITOR AS T2 ON T1.visitor_ID   =   T2.ID WHERE T2.ID   =   5 \n    ```````````\n    \"\"\"\n\nuser_input = \"Table car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nmessages = [{\n        \"role\": \"system\",\n        \"content\": system_prompt\n    },\n    {\n        \"role\": \"user\",\n        \"content\": user_input\n    }\n]\n\ncompletion = client.chat.completions.create(\n    model=\"gpt-4-turbo-preview\",\n    messages=messages,\n    temperature=0.7,\n    n=5\n)\n\nfor choice in completion.choices:\n    print(choice.message.content + \"\\n\")\n\n```\n\n    Q: What is the average horsepower for cars made in Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ffee6f55-2362-4906-aae9-a7e497df2526
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ffee6f55-2362-4906-aae9-a7e497df2526\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1b4b880-25ec-4307-ba60-f1304aed71e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"85b4c5371c3bb17dbc8099bd6eaa26b3a14eff3e793ef654b45c38fe7e84f368\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b11368cbd5e605f39b7c48292cbf7cce0831e71d58e51a566ce46279fe839d9b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11601, \"end_char_idx\": 13747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\n    \n    Q: What is the average horsepower for cars made by makers from Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    \n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n\n\nOnce we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ffee6f55-2362-4906-aae9-a7e497df2526\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0006e3efd6d39948546e561c9e61c61cd06b236878d5d6f4e43191a77590f4ef\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ca912825-4eeb-4b31-bafc-8d9341952498\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b450c9ac0bbdae406b9891bad4de9e27a58db3b165a5b0f405ba1c9eb765058\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12037, \"end_char_idx\": 14979, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\n    \n    Q: What is the average horsepower for cars made by makers from Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    \n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n\n\nOnce we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nfor choice in completion.choices:\n    question = choice.message.content.split(\"Q: \")[1].split(\"\\n\")[0]  # Extracting the question\n    answer = choice.message.content.split(\"\\nA: \")[1].split(\"\\n\")[0]  # Extracting the answer\n    eval_data.append({\n        \"input\": [\n            {\"role\": \"system\", \"content\": input_prompt},\n            {\"role\": \"user\", \"content\": question},\n        ],\n        \"ideal\": answer\n    })\n\nfor item in eval_data:\n    print(item)\n```\n\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ca912825-4eeb-4b31-bafc-8d9341952498
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ca912825-4eeb-4b31-bafc-8d9341952498\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4ecc462842f51fa1d62c7b941ef819b649c02517db7cc39dcda6c4af175edb64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9050baf6-de93-4d88-969c-ea8a95c859f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5ac2d4cdb10c785db2011cd46194ded7b48ac4d8a3a3804c2a043b8e3ba77aea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13534, \"end_char_idx\": 16212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Once we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nfor choice in completion.choices:\n    question = choice.message.content.split(\"Q: \")[1].split(\"\\n\")[0]  # Extracting the question\n    answer = choice.message.content.split(\"\\nA: \")[1].split(\"\\n\")[0]  # Extracting the answer\n    eval_data.append({\n        \"input\": [\n            {\"role\": \"system\", \"content\": input_prompt},\n            {\"role\": \"user\", \"content\": question},\n        ],\n        \"ideal\": answer\n    })\n\nfor item in eval_data:\n    print(item)\n```\n\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9050baf6-de93-4d88-969c-ea8a95c859f3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9050baf6-de93-4d88-969c-ea8a95c859f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ca912825-4eeb-4b31-bafc-8d9341952498\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"17f2f92d33fb1019cb102f93322912962622a079e9ed77b3e14c07722c20a845\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4c45bd2-42f4-41df-b807-8337df817acb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0759faa0866775bc7fd049a62d7ebf20e2e96bcb9f6eb5e0036a653002f4b05d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14980, \"end_char_idx\": 17331, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c4c45bd2-42f4-41df-b807-8337df817acb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c4c45bd2-42f4-41df-b807-8337df817acb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9050baf6-de93-4d88-969c-ea8a95c859f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5930df01517ce71a58f03767cf0aa626db1a5ed0b356aac526e6c24108b816c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"65c324e3-1965-446a-b52a-a13aad4af440\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"feedee288015d6864fb34882ead95610e365e753a7191adbaffdf52914fa3c2a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16213, \"end_char_idx\": 18598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': \"What is the average horsepower for cars produced in countries from the continent with the id '3'?\"}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 65c324e3-1965-446a-b52a-a13aad4af440
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"65c324e3-1965-446a-b52a-a13aad4af440\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4c45bd2-42f4-41df-b807-8337df817acb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4137499b1e28ebd3cc5afd3c435fa2cce5bc70d8077928be5c94edfcd56f88d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bbada030-e7cc-4cce-b619-b8f615b33985\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2eeb961643986f96eaa89dfd6bcce8bc5c156ce5ebd6299b46993006b5d6c333\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17332, \"end_char_idx\": 19843, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': \"What is the average horsepower for cars produced in countries from the continent with the id '3'?\"}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bbada030-e7cc-4cce-b619-b8f615b33985
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bbada030-e7cc-4cce-b619-b8f615b33985\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"65c324e3-1965-446a-b52a-a13aad4af440\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ed1f78af58ea25d5906f91b13cda30ca05735ebedb6b278d04580cca00235d5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"490c577a-fd2a-450f-8990-ac4dc95ee8ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f4101e8d4f54af493be4f6b69555c21d9f985f5a3be8fbfb485ad596ca6c88e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18599, \"end_char_idx\": 21288, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n\n\nNext we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 490c577a-fd2a-450f-8990-ac4dc95ee8ec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"490c577a-fd2a-450f-8990-ac4dc95ee8ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bbada030-e7cc-4cce-b619-b8f615b33985\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c0bab94aea84fbc8e14088a14d648be4d5e01de3895b0e280bdc9e79d6213c1a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"21ed9e72-8f48-4163-95e8-c86469109912\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"056e66da09a52a90c9b9f0370fe26e54477be31ae1e67d339493a333e7063741\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19844, \"end_char_idx\": 22364, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n\n\nNext we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:\n\n\n```python\n\"\"\"\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. \\\"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\\\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  \"\"\"\"\"\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 21ed9e72-8f48-4163-95e8-c86469109912
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"21ed9e72-8f48-4163-95e8-c86469109912\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"490c577a-fd2a-450f-8990-ac4dc95ee8ec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7dd70446cef1ed5aae510c0af2023c004dd7f661f1d09966a7cba2f1ae78bdae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"58096600-d1f9-4f21-b7fa-def0ad48b1f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"916cae0985b0e2eb687e7d89eed355ea3537abfedd50a5ef0e6545f213fbf667\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20844, \"end_char_idx\": 23437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Next we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:\n\n\n```python\n\"\"\"\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. \\\"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\\\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  \"\"\"\"\"\n```\n\n\n\n\n    '\\nspider-sql:\\n  id: spider-sql.dev.v0\\n  metrics: [accuracy]\\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\\n    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\\nspider-sql.dev.v0:\\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\\n  args:\\n    samples_jsonl: sql/spider_sql.jsonl\\n    eval_type: cot_classify\\n    modelgraded_spec: sql\\n  '"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 58096600-d1f9-4f21-b7fa-def0ad48b1f1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"58096600-d1f9-4f21-b7fa-def0ad48b1f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21ed9e72-8f48-4163-95e8-c86469109912\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"73f9b4e79a92391ad24e05b60f3af73836c371131cb0f0bed589bf2f1268e78f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"faad21f6-4e87-4232-ba9e-c08e26d73b43\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c88731d5079dd1904859263f84017b8c722a22fd922d538a3b3b88189dd88dbc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22373, \"end_char_idx\": 25130, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "'\\nspider-sql:\\n  id: spider-sql.dev.v0\\n  metrics: [accuracy]\\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\\n    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\\nspider-sql.dev.v0:\\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\\n  args:\\n    samples_jsonl: sql/spider_sql.jsonl\\n    eval_type: cot_classify\\n    modelgraded_spec: sql\\n  '\n\n\n\n## Running an evaluation\n\nWe can run this eval using the `oaieval` CLI. To get setup, install the library: `pip install .` (if you are running the [OpenAI Evals library](github.com/openai/evals) locally) or `pip install oaieval` if you are running an existing eval.\n\nThen, run the eval using the CLI: `oaieval gpt-3.5-turbo spider-sql`\n\nThis command expects a model name and an eval set name. Note that we provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals. The valid eval names are specified in the YAML files under `evals/registry/evals` and their corresponding implementations can be found in `evals/elsuite`.\n\n\n```python\n!pip install evals --quiet\n```\n\nThe `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\nThese CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\n`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).\n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: faad21f6-4e87-4232-ba9e-c08e26d73b43
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"faad21f6-4e87-4232-ba9e-c08e26d73b43\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"58096600-d1f9-4f21-b7fa-def0ad48b1f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6cb52d561c1c1f6423e78ae5ef41ce2c8e8983dcd7391c47aff9f6bd4bedb4f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fbc0c41e-25f5-4748-a681-8feb1ce0eb73\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0cafe37e6073396b41848498fbb85c6ebd20393fd8494a86f449b6b5ace42e5c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24129, \"end_char_idx\": 26532, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "```python\n!pip install evals --quiet\n```\n\nThe `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\nThese CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\n`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).\n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\n\n```python\n!oaieval gpt-3.5-turbo spider-sql --max_samples 25\n```\n\n    [2024-03-26 19:44:39,836] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/evals\n    [2024-03-26 19:44:43,623] [registry.py:257] Loading registry from /Users/shyamal/.evals/evals\n    [2024-03-26 19:44:43,635] [oaieval.py:189] \u001b[1;35mRun started: 240327024443FACXGMKA\u001b[0m\n    [2024-03-26 19:44:43,663] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/modelgraded\n    [2024-03-26 19:44:43,851] [registry.py:257] Loading registry from /Users/shyamal/.evals/modelgraded\n    [2024-03-26 19:44:43,853] [data.py:90] Fetching /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/data/sql/spider_sql.jsonl\n    [2024-03-26 19:44:43,878] [eval.py:36] Evaluating 25 samples\n    [2024-03-26 19:44:43,952] [eval.py:144] Running in threaded mode with 10 threads!\n      0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fbc0c41e-25f5-4748-a681-8feb1ce0eb73
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fbc0c41e-25f5-4748-a681-8feb1ce0eb73\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"faad21f6-4e87-4232-ba9e-c08e26d73b43\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"96f3c442809380861c315b93b399e78554300a75b29623c9a19337f61df53596\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d51222c7-d468-4d0e-81ae-b44b0d27ed13\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"081235968d0e797ca990c2399a521982c7e437b5f730de75a4af4662fb794ee5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26123, \"end_char_idx\": 28011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,090] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,145] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,971] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,040] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      4%|\u2588\u258a                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      8%|\u2588\u2588\u2588\u258c                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d51222c7-d468-4d0e-81ae-b44b0d27ed13
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d51222c7-d468-4d0e-81ae-b44b0d27ed13\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fbc0c41e-25f5-4748-a681-8feb1ce0eb73\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"11c6361da9251d3cd2ba3282aa24d66b0ced13e87bbd0da3075692a89caaa2ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3360e5d6-ec02-4ae9-ba85-51b94b6ee184\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db366f425162c8b26059e702eeec766dcdd8d5f223f0f4222a143afb335fa1f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27044, \"end_char_idx\": 28926, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      4%|\u2588\u258a                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      8%|\u2588\u2588\u2588\u258c                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     12%|\u2588\u2588\u2588\u2588\u2588\u258e                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     16%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     20%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3360e5d6-ec02-4ae9-ba85-51b94b6ee184
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3360e5d6-ec02-4ae9-ba85-51b94b6ee184\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d51222c7-d468-4d0e-81ae-b44b0d27ed13\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f8ce2ab67d930b434fb8e8540ec8fa019a319b00d39a9967f8a340969a8d2b54\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa23da19-833a-4999-a27c-2cb257a75f8f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57a75db204086cf2fa4fe1c71627c2c33d8915580d7201ceb300d6824ea63b00\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27972, \"end_char_idx\": 29854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     12%|\u2588\u2588\u2588\u2588\u2588\u258e                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     16%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     20%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     24%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     28%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fa23da19-833a-4999-a27c-2cb257a75f8f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fa23da19-833a-4999-a27c-2cb257a75f8f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3360e5d6-ec02-4ae9-ba85-51b94b6ee184\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"117aad98c637725c9d4a82276ba547a06c92f677ef9f87951c0ab4efbaa38f65\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c76b0d3-0e9a-4888-954f-19ebc1eeea09\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"51de4367ae40c6512905d30f1b3b99adce02a0a291ab512d027c14ca99febed4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28891, \"end_char_idx\": 30747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "1 200 OK\"\n    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     24%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     28%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     40%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     44%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6c76b0d3-0e9a-4888-954f-19ebc1eeea09
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6c76b0d3-0e9a-4888-954f-19ebc1eeea09\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa23da19-833a-4999-a27c-2cb257a75f8f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"70afdefdc12f4a25bffe98095dc610d0afa19c92b566d94a26e8c90869710122\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1d015f1-7e8a-4221-9241-0d99ba9be9f2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8306eec8985f16e0a9a4ee47cd9a2ccd16d9aff13d9cbd82a093be64dd1e450\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29780, \"end_char_idx\": 31636, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     40%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     44%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     52%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     56%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a1d015f1-7e8a-4221-9241-0d99ba9be9f2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a1d015f1-7e8a-4221-9241-0d99ba9be9f2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c76b0d3-0e9a-4888-954f-19ebc1eeea09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e72051191e6ef2aa1deb6f9c5d5a8fa74ed70bd12f05786c8ad5f3d119a83f4c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e7b3091a-e59c-489b-935e-d8f45d36dabb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f901a8ee5ed0a6397597abe944a5b433ddd0c7cc86e71208e4597435b045557\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30668, \"end_char_idx\": 32510, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     52%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     56%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     60%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     68%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e7b3091a-e59c-489b-935e-d8f45d36dabb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e7b3091a-e59c-489b-935e-d8f45d36dabb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1d015f1-7e8a-4221-9241-0d99ba9be9f2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"fa8c0206957e6eae8970e32ae1468b8ec5d06669c321b0910b282255130b2bc9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bc298cf6-d6bd-49e6-8ad5-f75c6094042c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"202c11a22b026984c25fa6e1b160ff245da11a8bbfe0356ecfc1b605cd17d31d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31582, \"end_char_idx\": 33355, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     60%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     68%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     76%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bc298cf6-d6bd-49e6-8ad5-f75c6094042c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bc298cf6-d6bd-49e6-8ad5-f75c6094042c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e7b3091a-e59c-489b-935e-d8f45d36dabb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"efab0118c51042a335009b590a6955420d360abfde5c8627b6edb800ec14f3e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3b7aaed0-0821-4e35-86ea-7cc8452c1d8e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5e494782afe4346ea938d8e0449b6bfb661e6650f10622756d2e34a12b0ebc0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32436, \"end_char_idx\": 34187, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     76%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     96%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3b7aaed0-0821-4e35-86ea-7cc8452c1d8e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3b7aaed0-0821-4e35-86ea-7cc8452c1d8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bc298cf6-d6bd-49e6-8ad5-f75c6094042c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c130447072542527b6981999946bfdecec4a0d9ba19faae2f9cb50138feeef6e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8dc6f031-0777-467d-8f2b-92a73a58a082\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d68329986a0dde70d712bd8b52a4af48e59c7b1f51214cff0084da94d46b8da1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33262, \"end_char_idx\": 34892, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     96%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25/25 [00:15<00:00,  1.60it/s]\n    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8dc6f031-0777-467d-8f2b-92a73a58a082
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8dc6f031-0777-467d-8f2b-92a73a58a082\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3b7aaed0-0821-4e35-86ea-7cc8452c1d8e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"261b701cd3da7888fbe90afbe22213be287e37f57e05c2372ed69737f59ac1a4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"417fb6de-d5a5-4a36-90b5-f47bf9fd58c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c76d41267ae2a9651a44e6c00718bfcfba021096ff7b65c123e3641062e87d15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34064, \"end_char_idx\": 35381, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25/25 [00:15<00:00,  1.60it/s]\n    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms\n\n\n`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.\n\n### Going through eval logs\n\nThe eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. \n\n\n```python\nlog_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # \"EDIT THIS\" - copy from above\nevents = f\"/tmp/evallogs/{log_name}\"\ndisplay(pd.read_json(events, lines=True).head(5))\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 417fb6de-d5a5-4a36-90b5-f47bf9fd58c8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"417fb6de-d5a5-4a36-90b5-f47bf9fd58c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8dc6f031-0777-467d-8f2b-92a73a58a082\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0789b1ccbe72dec2dcc31388d0b9f2df2fb905e9fb9c4ae493b33814081ce585\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"afd87d68-113c-4437-ab27-c6f502ec63a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df8d0a1d4d691c842a849196abc154b427afc611a470efec6e5c4ec9d7ad1557\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34307, \"end_char_idx\": 37478, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms\n\n\n`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.\n\n### Going through eval logs\n\nThe eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. \n\n\n```python\nlog_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # \"EDIT THIS\" - copy from above\nevents = f\"/tmp/evallogs/{log_name}\"\ndisplay(pd.read_json(events, lines=True).head(5))\n```\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>spec</th>\n      <th>final_report</th>\n      <th>run_id</th>\n      <th>event_id</th>\n      <th>sample_id</th>\n      <th>type</th>\n      <th>data</th>\n      <th>created_by</th>\n      <th>created_at</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>{'completion_fns': ['gpt-3.5-turbo'], 'eval_name': 'spider-sql.dev.v0', 'base_eval': 'spider-sql', 'split': 'dev', 'run_config': {'completion_fns': ['gpt-3.5-turbo'], 'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify', 'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry', 'args': {'samples_jsonl': 'sql/spider_sql.jsonl', 'eval_type': 'cot_classify', 'modelgraded_spec': 'sql'}, 'key': 'spider-sql.dev.v0', 'group': 'sql'}, 'seed': 20220722, 'max_samples': 25, 'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25', 'initial_settings': {'visible': False}}, 'created_by': '', 'run_id': '240327024443FACXGMKA', 'created_at': '2024-03-27 02:44:43.626043'}</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaT</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>NaN</td>\n      <td>{'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaT</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>0.0</td>\n      <td>spider-sql.dev.88</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: afd87d68-113c-4437-ab27-c6f502ec63a8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"afd87d68-113c-4437-ab27-c6f502ec63a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"417fb6de-d5a5-4a36-90b5-f47bf9fd58c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6653a273dcd5ea8f7c8b99df5e5dc9da02b8fa2281a5448872f66faccbb384eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3abbdfa-00b7-4db9-8c2c-57a613f96f35\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8fd03c3cad930b21286a3d08a17e04d228d35937cc112ef200190f164b83a650\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37479, \"end_char_idx\": 40052, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the average rank of winners in all matches.\n', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners\nFROM matches;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.821110+00:00</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>1.0</td>\n      <td>spider-sql.dev.82</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d3abbdfa-00b7-4db9-8c2c-57a613f96f35
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d3abbdfa-00b7-4db9-8c2c-57a613f96f35\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"afd87d68-113c-4437-ab27-c6f502ec63a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8e0cc7c740d25642b408b58fe3bc7fd3ec8cfa7f75b0e496cc34483c8da52c94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c3262d22-c2f9-4791-a866-e24d7e04f79b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ec710541b81819af80b8e7dbcb45936042d6b4ae46b4f832dc9a5bc72cec976\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39005, \"end_char_idx\": 41789, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS total_matches\nFROM matches;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.831848+00:00</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>2.0</td>\n      <td>spider-sql.dev.25</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.996647+00:00</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n```python\n# processing the log events generated by oaieval\n\nwith open(events, \"r\") as f:\n    events_df = pd.read_json(f, lines=True)\n```\n\nThis file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator\u2019s name, run ID, and creation timestamp.\n\n\n```python\ndisplay(events_df.iloc[0].spec)\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c3262d22-c2f9-4791-a866-e24d7e04f79b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c3262d22-c2f9-4791-a866-e24d7e04f79b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3abbdfa-00b7-4db9-8c2c-57a613f96f35\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0aa72862556fd4d1087ed1e904559866eb7b1ed33010f83d36c963b3f7b2ba7a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d90657ad-654d-4f84-adaf-789d37b373a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ec1f596b22aaf971bfc5a8f8ff999c198dc0be20e825f80bc68689950cad9b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 40498, \"end_char_idx\": 43047, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.996647+00:00</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n```python\n# processing the log events generated by oaieval\n\nwith open(events, \"r\") as f:\n    events_df = pd.read_json(f, lines=True)\n```\n\nThis file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator\u2019s name, run ID, and creation timestamp.\n\n\n```python\ndisplay(events_df.iloc[0].spec)\n```\n\n\n    {'completion_fns': ['gpt-3.5-turbo'],\n     'eval_name': 'spider-sql.dev.v0',\n     'base_eval': 'spider-sql',\n     'split': 'dev',\n     'run_config': {'completion_fns': ['gpt-3.5-turbo'],\n      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',\n       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',\n       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',\n        'eval_type': 'cot_classify',\n        'modelgraded_spec': 'sql'},\n       'key': 'spider-sql.dev.v0',\n       'group': 'sql'},\n      'seed': 20220722,\n      'max_samples': 25,\n      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',\n      'initial_settings': {'visible': False}},\n     'created_by': '',\n     'run_id': '240327024443FACXGMKA',\n     'created_at': '2024-03-27 02:44:43.626043'}\n\n\nLet's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d90657ad-654d-4f84-adaf-789d37b373a4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d90657ad-654d-4f84-adaf-789d37b373a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c3262d22-c2f9-4791-a866-e24d7e04f79b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8bf7e16f58920fd038da862bae8d9519166af984e5674ee69520d94badf1102b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9d7cbc01-d5c7-449b-8e82-9ad10a83a90c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b13c7ab4b4dfd52de72c9adbe24ae0c4bcfb57d878aff96a2106c02a6abfea0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41744, \"end_char_idx\": 43229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "```python\ndisplay(events_df.iloc[0].spec)\n```\n\n\n    {'completion_fns': ['gpt-3.5-turbo'],\n     'eval_name': 'spider-sql.dev.v0',\n     'base_eval': 'spider-sql',\n     'split': 'dev',\n     'run_config': {'completion_fns': ['gpt-3.5-turbo'],\n      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',\n       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',\n       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',\n        'eval_type': 'cot_classify',\n        'modelgraded_spec': 'sql'},\n       'key': 'spider-sql.dev.v0',\n       'group': 'sql'},\n      'seed': 20220722,\n      'max_samples': 25,\n      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',\n      'initial_settings': {'visible': False}},\n     'created_by': '',\n     'run_id': '240327024443FACXGMKA',\n     'created_at': '2024-03-27 02:44:43.626043'}\n\n\nLet's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.\n\n\n```python\npd.set_option('display.max_colwidth', None)  # None means no truncation\ndisplay(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9d7cbc01-d5c7-449b-8e82-9ad10a83a90c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9d7cbc01-d5c7-449b-8e82-9ad10a83a90c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d90657ad-654d-4f84-adaf-789d37b373a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c51f334d524e91795754f0f48a6969035394eb4997035234fc6564420e9b915c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"98a3e310-da47-47b5-8e62-14d853534075\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c4d59968cf169d38bc2b3a2d85d03618c48f1467be680f6a4ebb8c74c7a023e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 42680, \"end_char_idx\": 51567, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Let's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.\n\n\n```python\npd.set_option('display.max_colwidth', None)  # None means no truncation\ndisplay(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])\n```\n\n\n    run_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                240327024443FACXGMKA\n    event_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               0.0\n    sample_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                spider-sql.dev.88\n    type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              sampling\n    data          {'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\n    Use only the following tables and columns:\n    Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\n    Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\n    Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n    \n    Question: Find the average rank of winners in all matches.\n    ', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners\n    FROM matches;']}\n    created_at                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                2024-03-27 02:44:44.821110+00:00\n    Name: 2, dtype: object\n\n\n\n```python\n# Inspect samples\nfor i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():\n    data = pd.json_normalize(row['data'])\n    print(f\"Prompt: {data['prompt'].iloc[0]}\")\n    print(f\"Sampled: {data['sampled'].iloc[0]}\")\n    print(\"-\" * 10)\n```\n\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 98a3e310-da47-47b5-8e62-14d853534075
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"98a3e310-da47-47b5-8e62-14d853534075\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9d7cbc01-d5c7-449b-8e82-9ad10a83a90c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6f579e2bac6958c8aa7072094195da82d0df93353548b4f117052193c79f76d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e399e844-2589-4cb0-816f-5104331be6b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d448681c028549f6f2c1d6c21014c4d15631cfa5b7cf63b8cf6cca01c6042648\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 50968, \"end_char_idx\": 53972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "```python\n# Inspect samples\nfor i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():\n    data = pd.json_normalize(row['data'])\n    print(f\"Prompt: {data['prompt'].iloc[0]}\")\n    print(f\"Sampled: {data['sampled'].iloc[0]}\")\n    print(\"-\" * 10)\n```\n\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the average rank of winners in all matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT AVG(winner_rank) AS average_rank_of_winners\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the total number of matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS total_matches\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e399e844-2589-4cb0-816f-5104331be6b0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e399e844-2589-4cb0-816f-5104331be6b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"98a3e310-da47-47b5-8e62-14d853534075\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"efe845d6cd0b8d48286c17dbf185fb910817e6237a605a9429e948392dee6c8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56ee54a0-0b89-4021-a602-ba0b238a7d5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8081cfa1c0edb27d083aaa3a7baaaeaa758676d384ae77990b417c063565201a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 52616, \"end_char_idx\": 55242, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the total number of matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS total_matches\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents. Columns: ContId (number), Continent (text)\\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\\n\\nQuestion: How many countries exist?\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS TotalCountries\\nFROM countries;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 56ee54a0-0b89-4021-a602-ba0b238a7d5f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"56ee54a0-0b89-4021-a602-ba0b238a7d5f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e399e844-2589-4cb0-816f-5104331be6b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2dac83b1bce5b1d9821abe9fdf90e512ba52cf62cf03ab3b89a73c7a7b827cb3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"abac6e26-f2ec-46f8-9506-f802acae65cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dad931d4882b288500dbac7b74da622f9de656e29875e150bb5538b7803bf14a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 53868, \"end_char_idx\": 56584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents. Columns: ContId (number), Continent (text)\\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\\n\\nQuestion: How many countries exist?\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS TotalCountries\\nFROM countries;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\\n\\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\\n', 'role': 'system'}]\n    Sampled: ['SELECT Title, Directed_by\\nFROM Cartoon\\nORDER BY Original_air_date;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\\n\\nQuestion: Show the name and the release year of the song by the youngest singer.\\n', 'role': 'system'}]\n    Sampled: ['```sql\\nSELECT s.Name, s.Song_release_year\\nFROM singer s\\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\\n```']\n    ----------\n\n\nLet's review our failures to understand which tests did not succeed."}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: abac6e26-f2ec-46f8-9506-f802acae65cb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"abac6e26-f2ec-46f8-9506-f802acae65cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56ee54a0-0b89-4021-a602-ba0b238a7d5f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"b759c9da2bf2f2536d413c263443f979c9667c39c375ef49fa37fac02e2c86a9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"32de8531-d78f-4cae-869b-643bfc2439a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b257916321e80c1f8c4b87ad84d80904cdb749c8f3349e8efb261dd3cf7dc6e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 55058, \"end_char_idx\": 58210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\\n\\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\\n', 'role': 'system'}]\n    Sampled: ['SELECT Title, Directed_by\\nFROM Cartoon\\nORDER BY Original_air_date;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\\n\\nQuestion: Show the name and the release year of the song by the youngest singer.\\n', 'role': 'system'}]\n    Sampled: ['```sql\\nSELECT s.Name, s.Song_release_year\\nFROM singer s\\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\\n```']\n    ----------\n\n\nLet's review our failures to understand which tests did not succeed.\n\n\n```python\ndef pretty_print_text(prompt):\n    # Define markers for the sections\n    markers = {\n        \"question\": \"[Question]:\",\n        \"expert\": \"[Expert]:\",\n        \"submission\": \"[Submission]:\",\n        \"end\": \"[END DATA]\"\n    }\n    \n    # Function to extract text between markers\n    def extract_text(start_marker, end_marker):\n        start = prompt.find(start_marker) + len(start_marker)\n        end = prompt.find(end_marker)\n        text = prompt[start:end].strip()\n        if start_marker == markers[\"question\"]:\n            text = text.split(\"\\n\\nQuestion:\")[-1].strip() if \"\\n\\nQuestion:\" in text else text\n        elif start_marker == markers[\"submission\"]:\n            text = text.replace(\"```sql\", \"\").replace(\"```\", \"\").strip()\n        return text\n    \n    # Extracting text for each section\n    question_text = extract_text(markers[\"question\"], markers[\"expert\"])\n    expert_text = extract_text(markers[\"expert\"], markers[\"submission\"])\n    submission_text = extract_text(markers[\"submission\"], markers[\"end\"])\n    \n    # HTML color codes and formatting\n    colors = {\n        \"question\": '<span style=\"color: #0000FF;\">QUESTION:<br>', \n        \"expert\": '<span style=\"color: #008000;\">EXPECTED:<br>',  \n        \"submission\": '<span style=\"color: #FFA500;\">SUBMISSION:<br>' \n    }\n    color_end = '</span>'\n    \n    # Display each section with color\n    from IPython.display import display, HTML\n    display(HTML(f\"{colors['question']}{question_text}{color_end}\"))\n    display(HTML(f\"{colors['expert']}{expert_text}{color_end}\"))\n    display(HTML(f\"{colors['submission']}{submission_text}{color_end}\"))\n```"}


created_at: 2024-08-26 20:31:29
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 32de8531-d78f-4cae-869b-643bfc2439a7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"32de8531-d78f-4cae-869b-643bfc2439a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a7ec993c-08fc-4643-844a-c524bc7c3f16\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"abac6e26-f2ec-46f8-9506-f802acae65cb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"57266b63ea63aeababf4f2c80f873e1339a5029ffb143149a18054eb9f8dc622\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7848bf6c-3848-4c35-9fd0-f75b3c9bac4b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e9f577507627bfa8df69233d7f66af92445e05915055a1bce054270fc75cce6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 56516, \"end_char_idx\": 59647, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "ref_doc_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "chroma:document": "Let's review our failures to understand which tests did not succeed.\n\n\n```python\ndef pretty_print_text(prompt):\n    # Define markers for the sections\n    markers = {\n        \"question\": \"[Question]:\",\n        \"expert\": \"[Expert]:\",\n        \"submission\": \"[Submission]:\",\n        \"end\": \"[END DATA]\"\n    }\n    \n    # Function to extract text between markers\n    def extract_text(start_marker, end_marker):\n        start = prompt.find(start_marker) + len(start_marker)\n        end = prompt.find(end_marker)\n        text = prompt[start:end].strip()\n        if start_marker == markers[\"question\"]:\n            text = text.split(\"\\n\\nQuestion:\")[-1].strip() if \"\\n\\nQuestion:\" in text else text\n        elif start_marker == markers[\"submission\"]:\n            text = text.replace(\"```sql\", \"\").replace(\"```\", \"\").strip()\n        return text\n    \n    # Extracting text for each section\n    question_text = extract_text(markers[\"question\"], markers[\"expert\"])\n    expert_text = extract_text(markers[\"expert\"], markers[\"submission\"])\n    submission_text = extract_text(markers[\"submission\"], markers[\"end\"])\n    \n    # HTML color codes and formatting\n    colors = {\n        \"question\": '<span style=\"color: #0000FF;\">QUESTION:<br>', \n        \"expert\": '<span style=\"color: #008000;\">EXPECTED:<br>',  \n        \"submission\": '<span style=\"color: #FFA500;\">SUBMISSION:<br>' \n    }\n    color_end = '</span>'\n    \n    # Display each section with color\n    from IPython.display import display, HTML\n    display(HTML(f\"{colors['question']}{question_text}{color_end}\"))\n    display(HTML(f\"{colors['expert']}{expert_text}{color_end}\"))\n    display(HTML(f\"{colors['submission']}{submission_text}{color_end}\"))\n```\n\n\n```python\n# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect\nfor i, row in events_df[events_df['type'] == 'metrics'].iterrows():\n    if row['data']['choice'] == 'Incorrect':\n        # Get the previous row's data, which contains the prompt and the expected result\n        prev_row = events_df.iloc[i-1]\n        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else \"Prompt not available\"\n        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')\n        \n        # Current row's data will be the actual result\n        result = row['data'].get('result', 'Actual result not provided')\n        \n        pretty_print_text(prompt)\n        print(\"-\" * 40)\n```\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>How many countries have a republic as their form of government?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  \"Republic\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT COUNT(*) \nFROM country \nWHERE GovernmentForm LIKE '%Republic%'\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.\n\n************</span>"}


