# embeddings_queue

created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: da83c696-787c-4d0c-b42f-5e3bd3fccaed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb", "title": "Azure functions example", "tokens": 2044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"da83c696-787c-4d0c-b42f-5e3bd3fccaed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68824bc8-c4c1-466d-9de6-534fb461f1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7e7debb5cef7878f989720fdc7817614a73b8c67ed7d3ba8cc7d107d219cbd8e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ccc286d9-e635-4869-9e74-f72cb6f8cb42\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cab966979611e5832219ed0f63b01d7f986022326e2c24aafec8dac0c8a89a6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3196, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "ref_doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "chroma:document": "# Azure functions example\n\n> Note: There is a newer version of the openai library available. See https://github.com/openai/openai-python/discussions/742\n\nThis notebook shows how to use the function calling capability with the Azure OpenAI service. Functions allow a caller of chat completions to define capabilities that the model can use to extend its\nfunctionality into external tools and data sources.\n\nYou can read more about chat functions on OpenAI's blog: https://openai.com/blog/function-calling-and-other-api-updates\n\n**NOTE**: Chat functions require model versions beginning with gpt-4 and gpt-35-turbo's `-0613` labels. They are not supported by older versions of the models.\n\n## Setup\n\nFirst, we install the necessary dependencies.\n\n\n```python\n! pip install \"openai>=0.28.1,<1.0.0\"\n# (Optional) If you want to use Microsoft Active Directory\n! pip install azure-identity\n```\n\n\n```python\nimport os\nimport openai\n```\n\n\nAdditionally, to properly access the Azure OpenAI Service, we need to create the proper resources at the [Azure Portal](https://portal.azure.com) (you can check a detailed guide on how to do this in the [Microsoft Docs](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal))\n\nOnce the resource is created, the first thing we need to use is its endpoint. You can get the endpoint by looking at the *\"Keys and Endpoints\"* section under the *\"Resource Management\"* section. Having this, we will set up the SDK using this information:\n\n\n```python\nopenai.api_base = \"\" # Add your endpoint here\n\n# functions is only supported by the 2023-07-01-preview API version\nopenai.api_version = \"2023-07-01-preview\"\n```\n\n### Authentication\n\nThe Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials.\n\n\n```python\nuse_azure_active_directory = False\n```\n\n\n#### Authentication using API key\n\nTo set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))\n\n\n```python\nif not use_azure_active_directory:\n    openai.api_type = \"azure\"\n    openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n```\n\n> Note: In this example, we configured the library to use the Azure API by setting the variables in code. For development, consider setting the environment variables instead:\n\n```\nOPENAI_API_BASE\nOPENAI_API_KEY\nOPENAI_API_TYPE\nOPENAI_API_VERSION\n```\n\n#### Authentication using Microsoft Active Directory\nLet's now see how we can get a key via Microsoft Active Directory Authentication.\n\n\n```python\nfrom azure.identity import DefaultAzureCredential\n\nif use_azure_active_directory:\n    default_credential = DefaultAzureCredential()\n    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n\n    openai.api_type = \"azure_ad\"\n    openai.api_key = token.token\n```\n\nA token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ccc286d9-e635-4869-9e74-f72cb6f8cb42
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb", "title": "Azure functions example", "tokens": 2044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ccc286d9-e635-4869-9e74-f72cb6f8cb42\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68824bc8-c4c1-466d-9de6-534fb461f1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7e7debb5cef7878f989720fdc7817614a73b8c67ed7d3ba8cc7d107d219cbd8e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da83c696-787c-4d0c-b42f-5e3bd3fccaed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5fc9e7d8513e2b302aba6ce81a160142cb417c3c404fb7c50bf7f45011767383\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fc372180-462d-4fd4-94a6-29e8533eb9c5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2cbc1d9adfb444a369b0c3ea7ccc19e9a9cda83f0347d49447669e7517f21eb5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1515, \"end_char_idx\": 4624, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "ref_doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "chroma:document": "```python\nopenai.api_base = \"\" # Add your endpoint here\n\n# functions is only supported by the 2023-07-01-preview API version\nopenai.api_version = \"2023-07-01-preview\"\n```\n\n### Authentication\n\nThe Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials.\n\n\n```python\nuse_azure_active_directory = False\n```\n\n\n#### Authentication using API key\n\nTo set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))\n\n\n```python\nif not use_azure_active_directory:\n    openai.api_type = \"azure\"\n    openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n```\n\n> Note: In this example, we configured the library to use the Azure API by setting the variables in code. For development, consider setting the environment variables instead:\n\n```\nOPENAI_API_BASE\nOPENAI_API_KEY\nOPENAI_API_TYPE\nOPENAI_API_VERSION\n```\n\n#### Authentication using Microsoft Active Directory\nLet's now see how we can get a key via Microsoft Active Directory Authentication.\n\n\n```python\nfrom azure.identity import DefaultAzureCredential\n\nif use_azure_active_directory:\n    default_credential = DefaultAzureCredential()\n    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n\n    openai.api_type = \"azure_ad\"\n    openai.api_key = token.token\n```\n\nA token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:\n\n\n```python\nimport typing\nimport time\nimport requests\n\nif typing.TYPE_CHECKING:\n    from azure.core.credentials import TokenCredential\n\nclass TokenRefresh(requests.auth.AuthBase):\n\n    def __init__(self, credential: \"TokenCredential\", scopes: typing.List[str]) -> None:\n        self.credential = credential\n        self.scopes = scopes\n        self.cached_token: typing.Optional[str] = None\n\n    def __call__(self, req):\n        if not self.cached_token or self.cached_token.expires_on - time.time() < 300:\n            self.cached_token = self.credential.get_token(*self.scopes)\n        req.headers[\"Authorization\"] = f\"Bearer {self.cached_token.token}\"\n        return req\n\nif use_azure_active_directory:\n    session = requests.Session()\n    session.auth = TokenRefresh(default_credential, [\"https://cognitiveservices.azure.com/.default\"])\n\n    openai.requestssession = session\n```\n\n## Functions\n\nWith setup and authentication complete, you can now use functions with the Azure OpenAI service. This will be split into a few steps:\n\n1. Define the function(s)\n2. Pass function definition(s) into chat completions API\n3. Call function with arguments from the response\n4. Feed function response back into chat completions API\n\n#### 1. Define the function(s)\n\nA list of functions can be defined, each containing the name of the function, an optional description, and the parameters the function accepts (described as a JSON schema)."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: fc372180-462d-4fd4-94a6-29e8533eb9c5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb", "title": "Azure functions example", "tokens": 2044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"fc372180-462d-4fd4-94a6-29e8533eb9c5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68824bc8-c4c1-466d-9de6-534fb461f1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7e7debb5cef7878f989720fdc7817614a73b8c67ed7d3ba8cc7d107d219cbd8e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ccc286d9-e635-4869-9e74-f72cb6f8cb42\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0b78f883fe845206c7546f8da8e3e0d60f459a916cec21d19f68e1cca861f56a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ae3ac786-3010-4bac-9745-d842dc25175f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"106b005d31dd35c722d8fac365ebb0cfed417618d47c51e9fd0091cb47374b5b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3199, \"end_char_idx\": 5919, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "ref_doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "chroma:document": "```python\nimport typing\nimport time\nimport requests\n\nif typing.TYPE_CHECKING:\n    from azure.core.credentials import TokenCredential\n\nclass TokenRefresh(requests.auth.AuthBase):\n\n    def __init__(self, credential: \"TokenCredential\", scopes: typing.List[str]) -> None:\n        self.credential = credential\n        self.scopes = scopes\n        self.cached_token: typing.Optional[str] = None\n\n    def __call__(self, req):\n        if not self.cached_token or self.cached_token.expires_on - time.time() < 300:\n            self.cached_token = self.credential.get_token(*self.scopes)\n        req.headers[\"Authorization\"] = f\"Bearer {self.cached_token.token}\"\n        return req\n\nif use_azure_active_directory:\n    session = requests.Session()\n    session.auth = TokenRefresh(default_credential, [\"https://cognitiveservices.azure.com/.default\"])\n\n    openai.requestssession = session\n```\n\n## Functions\n\nWith setup and authentication complete, you can now use functions with the Azure OpenAI service. This will be split into a few steps:\n\n1. Define the function(s)\n2. Pass function definition(s) into chat completions API\n3. Call function with arguments from the response\n4. Feed function response back into chat completions API\n\n#### 1. Define the function(s)\n\nA list of functions can be defined, each containing the name of the function, an optional description, and the parameters the function accepts (described as a JSON schema).\n\n\n```python\nfunctions = [\n    {\n        \"name\": \"get_current_weather\",\n        \"description\": \"Get the current weather\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\n                    \"type\": \"string\",\n                    \"description\": \"The city and state, e.g. San Francisco, CA\",\n                },\n                \"format\": {\n                    \"type\": \"string\",\n                    \"enum\": [\"celsius\", \"fahrenheit\"],\n                    \"description\": \"The temperature unit to use. Infer this from the users location.\",\n                },\n            },\n            \"required\": [\"location\"],\n        },\n    }\n]\n```\n\n#### 2. Pass function definition(s) into chat completions API\n\nNow we can pass the function into the chat completions API. If the model determines it should call the function, a `finish_reason` of \"function_call\" will be populated on the choice and the details of which function to call and its arguments will be present in the `message`. Optionally, you can set the `function_call` keyword argument to force the model to call a particular function (e.g. `function_call={\"name\": get_current_weather}`). By default, this is set to `auto`, allowing the model to choose whether to call the function or not."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ae3ac786-3010-4bac-9745-d842dc25175f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb", "title": "Azure functions example", "tokens": 2044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ae3ac786-3010-4bac-9745-d842dc25175f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68824bc8-c4c1-466d-9de6-534fb461f1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7e7debb5cef7878f989720fdc7817614a73b8c67ed7d3ba8cc7d107d219cbd8e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fc372180-462d-4fd4-94a6-29e8533eb9c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f25eaf1490fb4916407364a146c0612760774e5fc610f4f301b97cca278af62e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94722239-b54a-4025-b5e3-233276fd55e1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69071f9436c7333db7729dcb321def546fc67549864ed47183659c53d3d3ba57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4627, \"end_char_idx\": 7777, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "ref_doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "chroma:document": "```python\nfunctions = [\n    {\n        \"name\": \"get_current_weather\",\n        \"description\": \"Get the current weather\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\n                    \"type\": \"string\",\n                    \"description\": \"The city and state, e.g. San Francisco, CA\",\n                },\n                \"format\": {\n                    \"type\": \"string\",\n                    \"enum\": [\"celsius\", \"fahrenheit\"],\n                    \"description\": \"The temperature unit to use. Infer this from the users location.\",\n                },\n            },\n            \"required\": [\"location\"],\n        },\n    }\n]\n```\n\n#### 2. Pass function definition(s) into chat completions API\n\nNow we can pass the function into the chat completions API. If the model determines it should call the function, a `finish_reason` of \"function_call\" will be populated on the choice and the details of which function to call and its arguments will be present in the `message`. Optionally, you can set the `function_call` keyword argument to force the model to call a particular function (e.g. `function_call={\"name\": get_current_weather}`). By default, this is set to `auto`, allowing the model to choose whether to call the function or not. \n\n\n```python\nmessages = [\n    {\"role\": \"system\", \"content\": \"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\"},\n    {\"role\": \"user\", \"content\": \"What's the weather like today in Seattle?\"}\n]\n\nchat_completion = openai.ChatCompletion.create(\n    deployment_id=\"gpt-35-turbo-0613\",\n    messages=messages,\n    functions=functions,\n)\nprint(chat_completion)\n```\n\n    {\n      \"choices\": [\n        {\n          \"content_filter_results\": {},\n          \"finish_reason\": \"function_call\",\n          \"index\": 0,\n          \"message\": {\n            \"function_call\": {\n              \"arguments\": \"{\\n  \\\"location\\\": \\\"Seattle, WA\\\"\\n}\",\n              \"name\": \"get_current_weather\"\n            },\n            \"role\": \"assistant\"\n          }\n        }\n      ],\n      \"created\": 1689702512,\n      \"id\": \"chatcmpl-7dj6GkYdM7Vw9eGn02bc2qqjN70Ps\",\n      \"model\": \"gpt-4\",\n      \"object\": \"chat.completion\",\n      \"prompt_annotations\": [\n        {\n          \"content_filter_results\": {\n            \"hate\": {\n              \"filtered\": false,\n              \"severity\": \"safe\"\n            },\n            \"self_harm\": {\n              \"filtered\": false,\n              \"severity\": \"safe\"\n            },\n            \"sexual\": {\n              \"filtered\": false,\n              \"severity\": \"safe\"\n            },\n            \"violence\": {\n              \"filtered\": false,\n              \"severity\": \"safe\"\n            }\n          },\n          \"prompt_index\": 0\n        }\n      ],\n      \"usage\": {\n        \"completion_tokens\": 18,\n        \"prompt_tokens\": 115,\n        \"total_tokens\": 133\n      }\n    }\n\n\n#### 3. Call function with arguments from the response\n\nThe name of the function call will be one that was provided initially and the arguments will include JSON matching the schema included in the function definition."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 94722239-b54a-4025-b5e3-233276fd55e1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb", "title": "Azure functions example", "tokens": 2044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"94722239-b54a-4025-b5e3-233276fd55e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"68824bc8-c4c1-466d-9de6-534fb461f1c6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7e7debb5cef7878f989720fdc7817614a73b8c67ed7d3ba8cc7d107d219cbd8e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ae3ac786-3010-4bac-9745-d842dc25175f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/functions.ipynb\", \"title\": \"Azure functions example\", \"tokens\": 2044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"18c0f2dbee318b8ba0f49c3a1b0e9008583a96fb9efc6eba51903b71051be1fb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7559, \"end_char_idx\": 9153, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "ref_doc_id": "68824bc8-c4c1-466d-9de6-534fb461f1c6", "chroma:document": "#### 3. Call function with arguments from the response\n\nThe name of the function call will be one that was provided initially and the arguments will include JSON matching the schema included in the function definition.\n\n\n```python\nimport json\n\ndef get_current_weather(request):\n    \"\"\"\n    This function is for illustrative purposes.\n    The location and unit should be used to determine weather\n    instead of returning a hardcoded response.\n    \"\"\"\n    location = request.get(\"location\")\n    unit = request.get(\"unit\")\n    return {\"temperature\": \"22\", \"unit\": \"celsius\", \"description\": \"Sunny\"}\n\nfunction_call =  chat_completion.choices[0].message.function_call\nprint(function_call.name)\nprint(function_call.arguments)\n\nif function_call.name == \"get_current_weather\":\n    response = get_current_weather(json.loads(function_call.arguments))\n```\n\n    get_current_weather\n    {\n      \"location\": \"Seattle, WA\"\n    }\n\n\n#### 4. Feed function response back into chat completions API\n\nThe response from the function should be serialized into a new message with the role set to \"function\". Now the model will use the response data to formulate its answer.\n\n\n```python\nmessages.append(\n    {\n        \"role\": \"function\",\n        \"name\": \"get_current_weather\",\n        \"content\": json.dumps(response)\n    }\n)\n\nfunction_completion = openai.ChatCompletion.create(\n    deployment_id=\"gpt-35-turbo-0613\",\n    messages=messages,\n    functions=functions,\n)\n\nprint(function_completion.choices[0].message.content.strip())\n```\n\n    Today in Seattle, the weather is sunny with a temperature of 22 degrees celsius."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7c58da77-bc6f-466b-a543-160642cb25dd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7c58da77-bc6f-466b-a543-160642cb25dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9fbe4aaa-68c0-482e-bb81-20e24d20e37a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"586b3250940add4ad34517f4615bf1d91659742f1862071264163349fce76328\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3277, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "# Azure chat completion models with your own data (preview)\n\n> Note: There is a newer version of the openai library available. See https://github.com/openai/openai-python/discussions/742\n\nThis example shows how to use Azure OpenAI service models with your own data. The feature is currently in preview. \n\nAzure OpenAI on your data enables you to run supported chat models such as GPT-3.5-Turbo and GPT-4 on your data without needing to train or fine-tune models. Running models on your data enables you to chat on top of, and analyze your data with greater accuracy and speed. One of the key benefits of Azure OpenAI on your data is its ability to tailor the content of conversational AI. Because the model has access to, and can reference specific sources to support its responses, answers are not only based on its pretrained knowledge but also on the latest information available in the designated data source. This grounding data also helps the model avoid generating responses based on outdated or incorrect information.\n\nAzure OpenAI on your own data with Azure Cognitive Search provides a customizable, pre-built solution for knowledge retrieval, from which a conversational AI application can be built. To see alternative methods for knowledge retrieval and semantic search, check out the cookbook examples for [vector databases](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases).\n\n## How it works\n\n[Azure OpenAI on your own data](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data) connects the model with your data, giving it the ability to retrieve and utilize data in a way that enhances the model's output. Together with Azure Cognitive Search, data is retrieved from designated data sources based on the user input and provided conversation history. The data is then augmented and resubmitted as a prompt to the model, giving the model contextual information it can use to generate a response.\n\nSee the [Data, privacy, and security for Azure OpenAI Service](https://learn.microsoft.com/legal/cognitive-services/openai/data-privacy?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext) for more information.\n\n## Prerequisites\nTo get started, we'll cover a few prequisites. \n\nTo properly access the Azure OpenAI Service, we need to create the proper resources at the [Azure Portal](https://portal.azure.com) (you can check a detailed guide on how to do this in the [Microsoft Docs](https://learn.microsoft.com/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal))\n\nTo use your own data with Azure OpenAI models, you will need:\n\n1. Azure OpenAI access and a resource with a chat model deployed (for example, GPT-3 or GPT-4)\n2. Azure Cognitive Search resource\n3. Azure Blob Storage resource\n4. Your documents to be used as data (See [data source options](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data#data-source-options))\n\n\nFor a full walk-through on how to upload your documents to blob storage and create an index using the Azure AI Studio, see this [Quickstart](https://learn.microsoft.com/azure/ai-services/openai/use-your-data-quickstart?pivots=programming-language-studio&tabs=command-line).\n\n## Setup\n\nFirst, we install the necessary dependencies."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9fbe4aaa-68c0-482e-bb81-20e24d20e37a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9fbe4aaa-68c0-482e-bb81-20e24d20e37a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c58da77-bc6f-466b-a543-160642cb25dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8429895381d96c71aa3f6af88d6eb9888d8e2bd5035adc2fd1b304b41a85a2b9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"42dd4b17-a227-4d04-9634-7946c56653f6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60c229b8b541dde9853e94bab547a578d8beb322acc34debe6fc771c0c7d4e93\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2947, \"end_char_idx\": 6078, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "For a full walk-through on how to upload your documents to blob storage and create an index using the Azure AI Studio, see this [Quickstart](https://learn.microsoft.com/azure/ai-services/openai/use-your-data-quickstart?pivots=programming-language-studio&tabs=command-line).\n\n## Setup\n\nFirst, we install the necessary dependencies.\n\n\n```python\n! pip install \"openai>=0.28.1,<1.0.0\"\n! pip install python-dotenv\n```\n\nIn this example, we'll use `dotenv` to load our environment variables. To connect with Azure OpenAI and the Search index, the following variables should be added to a `.env` file in `KEY=VALUE` format:\n\n* `OPENAI_API_BASE` - the Azure OpenAI endpoint. This can be found under \"Keys and Endpoints\" for your Azure OpenAI resource in the Azure Portal.\n* `OPENAI_API_KEY` - the Azure OpenAI API key. This can be found under \"Keys and Endpoints\" for your Azure OpenAI resource in the Azure Portal. Omit if using Azure Active Directory authentication (see below `Authentication using Microsoft Active Directory`)\n* `SEARCH_ENDPOINT` - the Cognitive Search endpoint. This URL be found on the \"Overview\" of your Search resource on the Azure Portal.\n* `SEARCH_KEY` - the Cognitive Search API key. Found under \"Keys\" for your Search resource in the Azure Portal.\n* `SEARCH_INDEX_NAME` - the name of the index you created with your own data.\n\n\n```python\nimport os\nimport openai\nimport dotenv\n\ndotenv.load_dotenv()\n```\n\n\n```python\nopenai.api_base = os.environ[\"OPENAI_API_BASE\"]\n\n# Azure OpenAI on your own data is only supported by the 2023-08-01-preview API version\nopenai.api_version = \"2023-08-01-preview\"\n```\n\n### Authentication\n\nThe Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials.\n\n\n```python\nuse_azure_active_directory = False  # Set this flag to True if you are using Azure Active Directory\n```\n\n\n#### Authentication using API key\n\nTo set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))\n\n\n```python\nif not use_azure_active_directory:\n    openai.api_type = 'azure'\n    openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n```\n\n#### Authentication using Microsoft Active Directory\nLet's now see how we can get a key via Microsoft Active Directory Authentication. See the [documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity) for more information on how to set this up.\n\n\n```python\n! pip install azure-identity\n```\n\n\n```python\nfrom azure.identity import DefaultAzureCredential\n\nif use_azure_active_directory:\n    default_credential = DefaultAzureCredential()\n    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n\n    openai.api_type = \"azure_ad\"\n    openai.api_key = token.token\n```\n\nA token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 42dd4b17-a227-4d04-9634-7946c56653f6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"42dd4b17-a227-4d04-9634-7946c56653f6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9fbe4aaa-68c0-482e-bb81-20e24d20e37a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e3c2bbb6c028a22a002ab3a5db2773bc4b265fc927c545cdc3dd0ebdb0c4333a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a9f89d7-da17-490c-bd97-330e9ecd41e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ebbd22a3e245485bf782c9cb56f7d1d42cf55e080354e847069f1f66e6e6ec1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4370, \"end_char_idx\": 7910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "```python\nopenai.api_base = os.environ[\"OPENAI_API_BASE\"]\n\n# Azure OpenAI on your own data is only supported by the 2023-08-01-preview API version\nopenai.api_version = \"2023-08-01-preview\"\n```\n\n### Authentication\n\nThe Azure OpenAI service supports multiple authentication mechanisms that include API keys and Azure credentials.\n\n\n```python\nuse_azure_active_directory = False  # Set this flag to True if you are using Azure Active Directory\n```\n\n\n#### Authentication using API key\n\nTo set up the OpenAI SDK to use an *Azure API Key*, we need to set up the `api_type` to `azure` and set `api_key` to a key associated with your endpoint (you can find this key in *\"Keys and Endpoints\"* under *\"Resource Management\"* in the [Azure Portal](https://portal.azure.com))\n\n\n```python\nif not use_azure_active_directory:\n    openai.api_type = 'azure'\n    openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n```\n\n#### Authentication using Microsoft Active Directory\nLet's now see how we can get a key via Microsoft Active Directory Authentication. See the [documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity) for more information on how to set this up.\n\n\n```python\n! pip install azure-identity\n```\n\n\n```python\nfrom azure.identity import DefaultAzureCredential\n\nif use_azure_active_directory:\n    default_credential = DefaultAzureCredential()\n    token = default_credential.get_token(\"https://cognitiveservices.azure.com/.default\")\n\n    openai.api_type = \"azure_ad\"\n    openai.api_key = token.token\n```\n\nA token is valid for a period of time, after which it will expire. To ensure a valid token is sent with every request, you can refresh an expiring token by hooking into requests.auth:\n\n\n```python\nimport typing\nimport time\nimport requests\n\nif typing.TYPE_CHECKING:\n    from azure.core.credentials import TokenCredential\n\nclass TokenRefresh(requests.auth.AuthBase):\n\n    def __init__(self, credential: \"TokenCredential\", scopes: typing.List[str]) -> None:\n        self.credential = credential\n        self.scopes = scopes\n        self.cached_token: typing.Optional[str] = None\n\n    def __call__(self, req):\n        if not self.cached_token or self.cached_token.expires_on - time.time() < 300:\n            self.cached_token = self.credential.get_token(*self.scopes)\n        req.headers[\"Authorization\"] = f\"Bearer {self.cached_token.token}\"\n        return req\n\n```\n\n## Chat completion model with your own data\n\n### Setting the context\n\nIn this example, we want our model to base its responses on Azure AI services documentation data. Following the [Quickstart](https://learn.microsoft.com/azure/ai-services/openai/use-your-data-quickstart?tabs=command-line&pivots=programming-language-studio) shared previously, we have added the [markdown](https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/ai-services/cognitive-services-and-machine-learning.md) file for the [Azure AI services and machine learning](https://learn.microsoft.com/azure/ai-services/cognitive-services-and-machine-learning) documentation page to our search index. The model is now ready to answer questions about Azure AI services and machine learning.\n\n### Code\n\nTo chat with Azure OpenAI models using your own data with the Python SDK, we must first set up the code to target the chat completions extensions endpoint which is designed to work with your own data. To do this, we've created a convenience function that can be called to set a custom adapter for the library which will target the extensions endpoint for a given deployment ID."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4a9f89d7-da17-490c-bd97-330e9ecd41e3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4a9f89d7-da17-490c-bd97-330e9ecd41e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"42dd4b17-a227-4d04-9634-7946c56653f6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0042dbb5409952fa1d21b717774eb79db92d41a8364790654248b918fb187cfe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5d8c6216-a9a6-4a14-97aa-5b4d5c4109c9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf79c03abacc8b2efda34e208cf91abda3608fc16bc4cc6b98c1db63f102f67a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6081, \"end_char_idx\": 9618, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "```python\nimport typing\nimport time\nimport requests\n\nif typing.TYPE_CHECKING:\n    from azure.core.credentials import TokenCredential\n\nclass TokenRefresh(requests.auth.AuthBase):\n\n    def __init__(self, credential: \"TokenCredential\", scopes: typing.List[str]) -> None:\n        self.credential = credential\n        self.scopes = scopes\n        self.cached_token: typing.Optional[str] = None\n\n    def __call__(self, req):\n        if not self.cached_token or self.cached_token.expires_on - time.time() < 300:\n            self.cached_token = self.credential.get_token(*self.scopes)\n        req.headers[\"Authorization\"] = f\"Bearer {self.cached_token.token}\"\n        return req\n\n```\n\n## Chat completion model with your own data\n\n### Setting the context\n\nIn this example, we want our model to base its responses on Azure AI services documentation data. Following the [Quickstart](https://learn.microsoft.com/azure/ai-services/openai/use-your-data-quickstart?tabs=command-line&pivots=programming-language-studio) shared previously, we have added the [markdown](https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/ai-services/cognitive-services-and-machine-learning.md) file for the [Azure AI services and machine learning](https://learn.microsoft.com/azure/ai-services/cognitive-services-and-machine-learning) documentation page to our search index. The model is now ready to answer questions about Azure AI services and machine learning.\n\n### Code\n\nTo chat with Azure OpenAI models using your own data with the Python SDK, we must first set up the code to target the chat completions extensions endpoint which is designed to work with your own data. To do this, we've created a convenience function that can be called to set a custom adapter for the library which will target the extensions endpoint for a given deployment ID.\n\n\n```python\nimport requests\n\ndef setup_byod(deployment_id: str) -> None:\n    \"\"\"Sets up the OpenAI Python SDK to use your own data for the chat endpoint.\n    \n    :param deployment_id: The deployment ID for the model to use with your own data.\n\n    To remove this configuration, simply set openai.requestssession to None.\n    \"\"\"\n\n    class BringYourOwnDataAdapter(requests.adapters.HTTPAdapter):\n\n        def send(self, request, **kwargs):\n            request.url = f\"{openai.api_base}/openai/deployments/{deployment_id}/extensions/chat/completions?api-version={openai.api_version}\"\n            return super().send(request, **kwargs)\n\n    session = requests.Session()\n\n    # Mount a custom adapter which will use the extensions endpoint for any call using the given `deployment_id`\n    session.mount(\n        prefix=f\"{openai.api_base}/openai/deployments/{deployment_id}\",\n        adapter=BringYourOwnDataAdapter()\n    )\n\n    if use_azure_active_directory:\n        session.auth = TokenRefresh(default_credential, [\"https://cognitiveservices.azure.com/.default\"])\n\n    openai.requestssession = session\n\n```\n\nNow we can call the convenience function to configure the SDK with the model we plan to use for our own data.\n\n\n```python\nsetup_byod(\"gpt-4\")\n```\n\nProviding our search endpoint, key, and index name for the `dataSources` keyword argument, any questions posed to the model will now be grounded in our own data. An additional property, `context`, will be provided to show the data the model referenced to answer the question.\n\n\n```python\ncompletion = openai.ChatCompletion.create(\n    messages=[{\"role\": \"user\", \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5d8c6216-a9a6-4a14-97aa-5b4d5c4109c9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5d8c6216-a9a6-4a14-97aa-5b4d5c4109c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a9f89d7-da17-490c-bd97-330e9ecd41e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"baca3b1176fc134b7d58844341a3be7eb3ae19a99784d311671fdc781fc26615\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"33561b03-4f57-47fa-af68-4cc3ae7d74fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b8d61ef0c1cb5187e0080b64927a23c58fad5d58d7588785081657fba1783888\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7913, \"end_char_idx\": 11185, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "```python\nimport requests\n\ndef setup_byod(deployment_id: str) -> None:\n    \"\"\"Sets up the OpenAI Python SDK to use your own data for the chat endpoint.\n    \n    :param deployment_id: The deployment ID for the model to use with your own data.\n\n    To remove this configuration, simply set openai.requestssession to None.\n    \"\"\"\n\n    class BringYourOwnDataAdapter(requests.adapters.HTTPAdapter):\n\n        def send(self, request, **kwargs):\n            request.url = f\"{openai.api_base}/openai/deployments/{deployment_id}/extensions/chat/completions?api-version={openai.api_version}\"\n            return super().send(request, **kwargs)\n\n    session = requests.Session()\n\n    # Mount a custom adapter which will use the extensions endpoint for any call using the given `deployment_id`\n    session.mount(\n        prefix=f\"{openai.api_base}/openai/deployments/{deployment_id}\",\n        adapter=BringYourOwnDataAdapter()\n    )\n\n    if use_azure_active_directory:\n        session.auth = TokenRefresh(default_credential, [\"https://cognitiveservices.azure.com/.default\"])\n\n    openai.requestssession = session\n\n```\n\nNow we can call the convenience function to configure the SDK with the model we plan to use for our own data.\n\n\n```python\nsetup_byod(\"gpt-4\")\n```\n\nProviding our search endpoint, key, and index name for the `dataSources` keyword argument, any questions posed to the model will now be grounded in our own data. An additional property, `context`, will be provided to show the data the model referenced to answer the question.\n\n\n```python\ncompletion = openai.ChatCompletion.create(\n    messages=[{\"role\": \"user\", \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?\"}],\n    deployment_id=\"gpt-4\",\n    dataSources=[  # camelCase is intentional, as this is the format the API expects\n        {\n            \"type\": \"AzureCognitiveSearch\",\n            \"parameters\": {\n                \"endpoint\": os.environ[\"SEARCH_ENDPOINT\"],\n                \"key\": os.environ[\"SEARCH_KEY\"],\n                \"indexName\": os.environ[\"SEARCH_INDEX_NAME\"],\n            }\n        }\n    ]\n)\nprint(completion)\n```\n\n    {\n      \"id\": \"65b485bb-b3c9-48da-8b6f-7d3a219f0b40\",\n      \"model\": \"gpt-4\",\n      \"created\": 1693338769,\n      \"object\": \"extensions.chat.completion\",\n      \"choices\": [\n        {\n          \"index\": 0,\n          \"finish_reason\": \"stop\",\n          \"message\": {\n            \"role\": \"assistant\",\n            \"content\": \"Azure AI services and Azure Machine Learning (AML) both aim to apply artificial intelligence (AI) to enhance business operations, but they target different audiences and offer different capabilities [doc1]. \\n\\nAzure AI services are designed for developers without machine learning experience and provide pre-trained models to solve general problems such as text analysis, image recognition, and natural language processing [doc5]. These services require general knowledge about your data without needing experience with machine learning or data science and provide REST APIs and language-based SDKs [doc2].\\n\\nOn the other hand, Azure Machine Learning is tailored for data scientists and involves a longer process of data collection, cleaning, transformation, algorithm selection, model training, and deployment [doc5]."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 33561b03-4f57-47fa-af68-4cc3ae7d74fb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"33561b03-4f57-47fa-af68-4cc3ae7d74fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5d8c6216-a9a6-4a14-97aa-5b4d5c4109c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a3ce9a46b5405c89e3299a5c7decded79f7bc2e35b8001c7a4fd2dff9c46fca5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"90e53513-51eb-4eb6-94a1-a76194be1acc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63046365a9134e4d1c9a9407f30b33e3057df13080bbcd1652981f2397624258\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9444, \"end_char_idx\": 12706, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "```python\ncompletion = openai.ChatCompletion.create(\n    messages=[{\"role\": \"user\", \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?\"}],\n    deployment_id=\"gpt-4\",\n    dataSources=[  # camelCase is intentional, as this is the format the API expects\n        {\n            \"type\": \"AzureCognitiveSearch\",\n            \"parameters\": {\n                \"endpoint\": os.environ[\"SEARCH_ENDPOINT\"],\n                \"key\": os.environ[\"SEARCH_KEY\"],\n                \"indexName\": os.environ[\"SEARCH_INDEX_NAME\"],\n            }\n        }\n    ]\n)\nprint(completion)\n```\n\n    {\n      \"id\": \"65b485bb-b3c9-48da-8b6f-7d3a219f0b40\",\n      \"model\": \"gpt-4\",\n      \"created\": 1693338769,\n      \"object\": \"extensions.chat.completion\",\n      \"choices\": [\n        {\n          \"index\": 0,\n          \"finish_reason\": \"stop\",\n          \"message\": {\n            \"role\": \"assistant\",\n            \"content\": \"Azure AI services and Azure Machine Learning (AML) both aim to apply artificial intelligence (AI) to enhance business operations, but they target different audiences and offer different capabilities [doc1]. \\n\\nAzure AI services are designed for developers without machine learning experience and provide pre-trained models to solve general problems such as text analysis, image recognition, and natural language processing [doc5]. These services require general knowledge about your data without needing experience with machine learning or data science and provide REST APIs and language-based SDKs [doc2].\\n\\nOn the other hand, Azure Machine Learning is tailored for data scientists and involves a longer process of data collection, cleaning, transformation, algorithm selection, model training, and deployment [doc5]. It allows users to create custom solutions for highly specialized and specific problems, requiring familiarity with the subject matter, data, and expertise in data science [doc5].\\n\\nIn summary, Azure AI services offer pre-trained models for developers without machine learning experience, while Azure Machine Learning is designed for data scientists to create custom solutions for specific problems.\",\n            \"end_turn\": true,\n            \"context\": {\n              \"messages\": [\n                {\n                  \"role\": \"tool\",\n                  \"content\": \"{\\\"citations\\\": [{\\\"content\\\": \\\"<h2 id=\\\\\\\"how-are-azure-ai-services-and-azure-machine-learning-aml-similar\\\\\\\">How are Azure AI services and Azure Machine Learning (AML) similar?.</h2>\\\\n<p>Both have the end-goal of applying artificial intelligence (AI) to enhance business operations, though how each provides this in the respective offerings is different..</p>\\\\n<p>Generally, the audiences are different:</p>\\\\n<ul>\\\\n<li>Azure AI services are for developers without machine-learning experience..</li>\\\\n<li>Azure Machine Learning is tailored for data scientists.\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 90e53513-51eb-4eb6-94a1-a76194be1acc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"90e53513-51eb-4eb6-94a1-a76194be1acc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"33561b03-4f57-47fa-af68-4cc3ae7d74fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2a32e0d3ef8893401ea306d3a55f7db1c5b8cc11694010ffc9225fdfdc7d0367\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bf817c34-2b13-47fa-bc70-28cd221ce39a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8979ad9c3242d54ffe7da0c085ba892f222e6254642de2eb126c9a64ae38fe8d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11186, \"end_char_idx\": 15018, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "It allows users to create custom solutions for highly specialized and specific problems, requiring familiarity with the subject matter, data, and expertise in data science [doc5].\\n\\nIn summary, Azure AI services offer pre-trained models for developers without machine learning experience, while Azure Machine Learning is designed for data scientists to create custom solutions for specific problems.\",\n            \"end_turn\": true,\n            \"context\": {\n              \"messages\": [\n                {\n                  \"role\": \"tool\",\n                  \"content\": \"{\\\"citations\\\": [{\\\"content\\\": \\\"<h2 id=\\\\\\\"how-are-azure-ai-services-and-azure-machine-learning-aml-similar\\\\\\\">How are Azure AI services and Azure Machine Learning (AML) similar?.</h2>\\\\n<p>Both have the end-goal of applying artificial intelligence (AI) to enhance business operations, though how each provides this in the respective offerings is different..</p>\\\\n<p>Generally, the audiences are different:</p>\\\\n<ul>\\\\n<li>Azure AI services are for developers without machine-learning experience..</li>\\\\n<li>Azure Machine Learning is tailored for data scientists.\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 2/Highlights=30 of size=137\\\"}, \\\"chunk_id\\\": \\\"2\\\"}, {\\\"content\\\": \\\"<td>Convert speech into text and text into natural-sounding speech..Translate from one language to another and enable speaker verification and recognition..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/vision/\\\\\\\">Vision</a></td>\\\\n<td>Recognize, identify, caption, index, and moderate your pictures, videos, and digital ink content..</td>\\\\n</tr>\\\\n<p></tbody>\\\\n</table></p>\\\\n<p>Use Azure AI services when you:</p>\\\\n<ul>\\\\n<li>Can use a generalized solution..</li>\\\\n</ul>\\\\n<p>Use other machine-learning solutions when you:</p>\\\\n<ul>\\\\n<li>Need to choose the algorithm and need to train on very specific data..</li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"what-is-machine-learning\\\\\\\">What is machine learning?.</h2>\\\\n<p>Machine learning is a concept where you bring together data and an algorithm to solve a specific need..Once the data and algorithm are trained, the output is a model that you can use again with different data..The trained model provides insights based on the new data..</p>\\\\n<p>The process of building a machine learning system requires some knowledge of machine learning or data science..</p>\\\\n<p>Machine learning is provided using <a href=\\\\\\\"/azure/architecture/data-guide/technology-choices/data-science-and-machine-learning?.context=azure%2fmachine-learning%2fstudio%2fcontext%2fml-context\\\\\\\">Azure Machine Learning (AML) products and services</a>..</p>\\\\n<h2 id=\\\\\\\"what-is-an-azure-ai-service\\\\\\\">What is an Azure AI service?.</h2>\\\\n<p>An Azure AI service provides part or all of the components in a machine learning solution: data, algorithm, and trained model..These services are meant to require general knowledge about your data without needing experience with machine learning or data science..These services provide both REST API(s) and language-based SDKs..As a result, you need to have programming language knowledge to use the services..</p>\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1188."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bf817c34-2b13-47fa-bc70-28cd221ce39a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bf817c34-2b13-47fa-bc70-28cd221ce39a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"90e53513-51eb-4eb6-94a1-a76194be1acc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cef0c84e6cf86ec28492405e6792d46154dd7f3c0516c241f24a356059a9f65f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa19e958-70d0-4e7d-be3f-615d17583ddb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7b49716a9ea6467ef0dd61a5fb6ae837adf9e75c4c59704c00af49d62d93feed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15019, \"end_char_idx\": 17882, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 1/Highlights=67 of size=506\\\"}, \\\"chunk_id\\\": \\\"1\\\"}, {\\\"content\\\": \\\"<hr />\\\\n<p>title: Azure AI services and Machine Learning\\\\ntitleSuffix: Azure AI services\\\\ndescription: Learn where Azure AI services fits in with other Azure offerings for machine learning.\\\\nservices: cognitive-services\\\\nmanager: nitinme\\\\nauthor: aahill\\\\nms.author: aahi\\\\nms.service: cognitive-services\\\\nms.topic: conceptual\\\\nms.date: 10/28/2021</p>\\\\n<hr />\\\\n<h1 id=\\\\\\\"azure-ai-services-and-machine-learning\\\\\\\">Azure AI services and machine learning</h1>\\\\n<p>Azure AI services provides machine learning capabilities to solve general problems such as analyzing text for emotional sentiment or analyzing images to recognize objects or faces..You don't need special machine learning or data science knowledge to use these services../what-are-ai-services.md\\\\\\\">Azure AI services</a> is a group of services, each supporting different, generalized prediction capabilities..The services are divided into different categories to help you find the right service..</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Service category</th>\\\\n<th>Purpose</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/decision/\\\\\\\">Decision</a></td>\\\\n<td>Build apps that surface recommendations for informed and efficient decision-making..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/lang/\\\\\\\">Language</a></td>\\\\n<td>Allow your apps to process natural language with pre-built scripts, evaluate sentiment and learn how to recognize what users want..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/search/\\\\\\\">Search</a></td>\\\\n<td>Add Bing Search APIs to your apps and harness the ability to comb billions of webpages, images, videos, and news with a single API call..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/speech/\\\\\\\">Speech</a></td>\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 0/Highlights=63 of size=526\\\"}, \\\"chunk_id\\\": \\\"0\\\"}, {\\\"content\\\": \\\"<p>How is Azure Cognitive Search related to Azure AI services?</p>\\\\n<p><a href=\\\\\\\"../search/search-what-is-azure-search.md\\\\\\\">Azure Cognitive Search</a> is a separate cloud search service that optionally uses Azure AI services to add image and natural language processing to indexing workloads."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: aa19e958-70d0-4e7d-be3f-615d17583ddb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"aa19e958-70d0-4e7d-be3f-615d17583ddb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf817c34-2b13-47fa-bc70-28cd221ce39a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4acb737234012803338ef9d747520fb6084c1b8e01a84bd855f1e185a27aa2da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"196e0763-cbf4-47d7-b991-8a63a7761302\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"31d0fc2d74f195d246d675dcf9a658ff9d584c8e090e32185482394725b7e5c9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17443, \"end_char_idx\": 20195, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 0/Highlights=63 of size=526\\\"}, \\\"chunk_id\\\": \\\"0\\\"}, {\\\"content\\\": \\\"<p>How is Azure Cognitive Search related to Azure AI services?</p>\\\\n<p><a href=\\\\\\\"../search/search-what-is-azure-search.md\\\\\\\">Azure Cognitive Search</a> is a separate cloud search service that optionally uses Azure AI services to add image and natural language processing to indexing workloads. Azure AI services is exposed in Azure Cognitive Search through <a href=\\\\\\\"../search/cognitive-search-predefined-skills.md\\\\\\\">built-in skills</a> that wrap individual APIs. You can use a free resource for walkthroughs, but plan on creating and attaching a <a href=\\\\\\\"../search/cognitive-search-attach-cognitive-services.md\\\\\\\">billable resource</a> for larger volumes.</p>\\\\n<h2 id=\\\\\\\"how-can-you-use-azure-ai-services\\\\\\\">How can you use Azure AI services?</h2>\\\\n<p>Each service provides information about your data. You can combine services together to chain solutions such as converting speech (audio) to text, translating the text into many languages, then using the translated languages to get answers from a knowledge base. While Azure AI services can be used to create intelligent solutions on their own, they can also be combined with traditional machine learning projects to supplement models or accelerate the development process. </p>\\\\n<p>Azure AI services that provide exported models for other machine learning tools:</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>Model information</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./custom-vision-service/overview.md\\\\\\\">Custom Vision</a></td>\\\\n<td><a href=\\\\\\\"./custom-vision-service/export-model-python.md\\\\\\\">Export</a> for Tensorflow for Android, CoreML for iOS11, ONNX for Windows ML</td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<h2 id=\\\\\\\"learn-more\\\\\\\">Learn more</h2>\\\\n<ul>\\\\n<li><a href=\\\\\\\"/azure/architecture/data-guide/technology-choices/data-science-and-machine-learning\\\\\\\">Architecture Guide - What are the machine learning products at Microsoft?</a></li>\\\\n<li><a href=\\\\\\\"../machine-learning/concept-deep-learning-vs-machine-learning.md\\\\\\\">Machine learning - Introduction to deep learning vs. machine learning</a></li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"next-steps\\\\\\\">Next steps</h2>\\\\n<ul>\\\\n<li>Create your Azure AI services resource in the <a href=\\\\\\\"multi-service-resource.md?pivots=azportal\\\\\\\">Azure portal</a> or with <a href=\\\\\\\"./multi-service-resource.md?pivots=azcli\\\\\\\">Azure CLI</a>.</li>\\\\n<li>Learn how to <a href=\\\\\\\"authentication.md\\\\\\\">authenticate</a> with your Azure AI service.</li>\\\\n<li>Use <a href=\\\\\\\"diagnostic-logging.md\\\\\\\">diagnostic logging</a> for issue identification and debugging."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 196e0763-cbf4-47d7-b991-8a63a7761302
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"196e0763-cbf4-47d7-b991-8a63a7761302\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa19e958-70d0-4e7d-be3f-615d17583ddb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7014fecb1a5fb758844430468d7b68bf63208cd2be3d2600a0ab7710818095d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"679c2369-0b87-4e2a-b410-7ca99833322b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b02ac08f4030b5348b2e9c7324f9b74bb33e58ac25ceebe006c9abf92dd1bf7a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "machine learning</a></li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"next-steps\\\\\\\">Next steps</h2>\\\\n<ul>\\\\n<li>Create your Azure AI services resource in the <a href=\\\\\\\"multi-service-resource.md?pivots=azportal\\\\\\\">Azure portal</a> or with <a href=\\\\\\\"./multi-service-resource.md?pivots=azcli\\\\\\\">Azure CLI</a>.</li>\\\\n<li>Learn how to <a href=\\\\\\\"authentication.md\\\\\\\">authenticate</a> with your Azure AI service.</li>\\\\n<li>Use <a href=\\\\\\\"diagnostic-logging.md\\\\\\\">diagnostic logging</a> for issue identification and debugging. </li>\\\\n<li>Deploy an Azure AI service in a Docker <a href=\\\\\\\"cognitive-services-container-support.md\\\\\\\">container</a>.</li>\\\\n<li>Keep up to date with <a href=\\\\\\\"https://azure.microsoft.com/updates/?product=cognitive-services\\\\\\\">service updates</a>.</li>\\\\n</ul>\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=793. Scores=3.3767838 and None.Org Highlight count=69.\\\"}, \\\"chunk_id\\\": \\\"3\\\"}, {\\\"content\\\": \\\"<p>How are Azure AI services different from machine learning?.</p>\\\\n<p>Azure AI services provide a trained model for you.This brings data and an algorithm together, available from a REST API(s) or SDK.An Azure AI service provides answers to general problems such as key phrases in text or item identification in images.</p>\\\\n<p>Machine learning is a process that generally requires a longer period of time to implement successfully.This time is spent on data collection, cleaning, transformation, algorithm selection, model training, and deployment to get to the same level of functionality provided by an Azure AI service.With machine learning, it is possible to provide answers to highly specialized and/or specific problems.Machine learning problems require familiarity with the specific subject matter and data of the problem under consideration, as well as expertise in data science.</p>\\\\n<h2 id=\\\\\\\"what-kind-of-data-do-you-have\\\\\\\">What kind of data do you have?.</h2>\\\\n<p>Azure AI services, as a group of services, can require none, some, or all custom data for the trained model.</p>\\\\n<h3 id=\\\\\\\"no-additional-training-data-required\\\\\\\">No additional training data required</h3>\\\\n<p>Services that provide a fully-trained model can be treated as a <em>opaque box</em>.You don't need to know how they work or what data was used to train them.</p>\\\\n<h3 id=\\\\\\\"some-or-all-training-data-required\\\\\\\">Some or all training data required</h3>\\\\n<p>Some services allow you to bring your own data, then train a model.This allows you to extend the model using the Service's data and algorithm with your own data.The output matches your needs.When you bring your own data, you may need to tag the data in a way specific to the service.For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 679c2369-0b87-4e2a-b410-7ca99833322b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"679c2369-0b87-4e2a-b410-7ca99833322b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"196e0763-cbf4-47d7-b991-8a63a7761302\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"00593b6246d27531f59a39580804bc12255c1953fa3d980d43091ad086fe3794\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c6da88a8-3f6e-4d28-851b-dcd295a349db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7bddc70942f09041900a29c8bcf0ae998ed248bbbd88d5bbda50751604565a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "available from a REST API(s) or SDK.An Azure AI service provides answers to general problems such as key phrases in text or item identification in images.</p>\\\\n<p>Machine learning is a process that generally requires a longer period of time to implement successfully.This time is spent on data collection, cleaning, transformation, algorithm selection, model training, and deployment to get to the same level of functionality provided by an Azure AI service.With machine learning, it is possible to provide answers to highly specialized and/or specific problems.Machine learning problems require familiarity with the specific subject matter and data of the problem under consideration, as well as expertise in data science.</p>\\\\n<h2 id=\\\\\\\"what-kind-of-data-do-you-have\\\\\\\">What kind of data do you have?.</h2>\\\\n<p>Azure AI services, as a group of services, can require none, some, or all custom data for the trained model.</p>\\\\n<h3 id=\\\\\\\"no-additional-training-data-required\\\\\\\">No additional training data required</h3>\\\\n<p>Services that provide a fully-trained model can be treated as a <em>opaque box</em>.You don't need to know how they work or what data was used to train them.</p>\\\\n<h3 id=\\\\\\\"some-or-all-training-data-required\\\\\\\">Some or all training data required</h3>\\\\n<p>Some services allow you to bring your own data, then train a model.This allows you to extend the model using the Service's data and algorithm with your own data.The output matches your needs.When you bring your own data, you may need to tag the data in a way specific to the service.For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model.These services process significant amounts of model data.</p>\\\\n<h2 id=\\\\\\\"service-requirements-for-the-data-model\\\\\\\">Service requirements for the data model</h2>\\\\n<p>The following data categorizes each service by which kind of data it allows or requires.</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>No training data required</th>\\\\n<th>You provide some or all training data</th>\\\\n<th>Real-time or near real-time data collection</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./LUIS/what-is-luis.md\\\\\\\">Language Understanding (LUIS)</a></td>\\\\n<td></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./personalizer/what-is-personalizer.md\\\\\\\">Personalizer</a><sup>1</sup></sup></td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./computer-vision/overview.md\\\\\\\">Vision</a></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n<td></td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<p><sup>1</sup> Personalizer only needs training data collected by the service (as it operates in real-time) to evaluate your policy and data."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c6da88a8-3f6e-4d28-851b-dcd295a349db
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c6da88a8-3f6e-4d28-851b-dcd295a349db\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"679c2369-0b87-4e2a-b410-7ca99833322b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7361747f4660ee85c249d17d742bb7160a04ca2b2f638c4c8b6213077f993f98\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"146a1dd0-9dea-4ec3-846d-f6f767485ccc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a14c3c2c76b52835756cd93ed995e5c5cfa322049edf65214ed8fb60705fa50\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model.These services process significant amounts of model data.</p>\\\\n<h2 id=\\\\\\\"service-requirements-for-the-data-model\\\\\\\">Service requirements for the data model</h2>\\\\n<p>The following data categorizes each service by which kind of data it allows or requires.</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>No training data required</th>\\\\n<th>You provide some or all training data</th>\\\\n<th>Real-time or near real-time data collection</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./LUIS/what-is-luis.md\\\\\\\">Language Understanding (LUIS)</a></td>\\\\n<td></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./personalizer/what-is-personalizer.md\\\\\\\">Personalizer</a><sup>1</sup></sup></td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./computer-vision/overview.md\\\\\\\">Vision</a></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n<td></td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<p><sup>1</sup> Personalizer only needs training data collected by the service (as it operates in real-time) to evaluate your policy and data.</p>\\\\n<h2 id=\\\\\\\"where-can-you-use-azure-ai-services\\\\\\\">Where can you use Azure AI services?.</h2>\\\\n<p>The services are used in any application that can make REST API(s) or SDK calls.Examples of applications include web sites, bots, virtual or mixed reality, desktop and mobile applications.\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1734. Scores=3.1447978 and None.Org Highlight count=66.Filtering to highlight size=891\\\"}, \\\"chunk_id\\\": \\\"4\\\"}], \\\"intent\\\": \\\"[\\\\\\\"What are the differences between Azure Machine Learning and Azure AI services?\\\\\\\"]\\\"}\",\n                  \"end_turn\": false\n                }\n              ]\n            }\n          }\n        }\n      ]\n    }\n\n\nIf you would prefer to stream the response from the model, you can pass the `stream=True` keyword argument:\n\n\n```python\nresponse = openai.ChatCompletion.create(\n    messages=[{\"role\": \"user\", \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 146a1dd0-9dea-4ec3-846d-f6f767485ccc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"146a1dd0-9dea-4ec3-846d-f6f767485ccc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c6da88a8-3f6e-4d28-851b-dcd295a349db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cd5f53380cce4334b1cde236f42cc9d848c3730b286e4d80443af61174d069ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"676909cb-6f06-455d-8edb-298195192d22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b1e286223839899cbd0ebf0b0c77aca33be11af8988cdba56fab51b0a4bf2c91\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "/computer-vision/overview.md\\\\\\\">Vision</a></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n<td></td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<p><sup>1</sup> Personalizer only needs training data collected by the service (as it operates in real-time) to evaluate your policy and data.</p>\\\\n<h2 id=\\\\\\\"where-can-you-use-azure-ai-services\\\\\\\">Where can you use Azure AI services?.</h2>\\\\n<p>The services are used in any application that can make REST API(s) or SDK calls.Examples of applications include web sites, bots, virtual or mixed reality, desktop and mobile applications.\\\", \\\"id\\\": null, \\\"title\\\": \\\"Azure AI services and machine learning\\\", \\\"filepath\\\": \\\"cognitive-services-and-machine-learning.md\\\", \\\"url\\\": \\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\", \\\"metadata\\\": {\\\"chunking\\\": \\\"orignal document size=1734. Scores=3.1447978 and None.Org Highlight count=66.Filtering to highlight size=891\\\"}, \\\"chunk_id\\\": \\\"4\\\"}], \\\"intent\\\": \\\"[\\\\\\\"What are the differences between Azure Machine Learning and Azure AI services?\\\\\\\"]\\\"}\",\n                  \"end_turn\": false\n                }\n              ]\n            }\n          }\n        }\n      ]\n    }\n\n\nIf you would prefer to stream the response from the model, you can pass the `stream=True` keyword argument:\n\n\n```python\nresponse = openai.ChatCompletion.create(\n    messages=[{\"role\": \"user\", \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?\"}],\n    deployment_id=\"gpt-4\",\n    dataSources=[\n        {\n            \"type\": \"AzureCognitiveSearch\",\n            \"parameters\": {\n                \"endpoint\": os.environ[\"SEARCH_ENDPOINT\"],\n                \"key\": os.environ[\"SEARCH_KEY\"],\n                \"indexName\": os.environ[\"SEARCH_INDEX_NAME\"],\n            }\n        }\n    ],\n    stream=True,\n)\n\nfor chunk in response:\n    delta = chunk.choices[0].delta\n\n    if \"role\" in delta:\n        print(\"\\n\"+ delta.role + \": \", end=\"\", flush=True)\n    if \"content\" in delta:\n        print(delta.content, end=\"\", flush=True)\n    if \"context\" in delta:\n        print(f\"Context: {delta.context}\", end=\"\", flush=True)\n```\n\n    Context: {\n      \"messages\": [\n        {\n          \"role\": \"tool\",\n          \"content\": \"{\\\"citations\\\":[{\\\"content\\\":\\\"<h2 id=\\\\\\\"how-are-azure-ai-services-and-azure-machine-learning-aml-similar\\\\\\\">How are Azure AI services and Azure Machine Learning (AML) similar?.</h2>\\\\n<p>Both have the end-goal of applying artificial intelligence (AI) to enhance business operations, though how each provides this in the respective offerings is different..</p>\\\\n<p>Generally, the audiences are different:</p>\\\\n<ul>\\\\n<li>Azure AI services are for developers without machine-learning experience..</li>\\\\n<li>Azure Machine Learning is tailored for data scientists.\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1188."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 676909cb-6f06-455d-8edb-298195192d22
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"676909cb-6f06-455d-8edb-298195192d22\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"146a1dd0-9dea-4ec3-846d-f6f767485ccc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02478c950afad54f966e7ba1c139891f5e109f036dce11caa3db629bb006ce37\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63eb6346-a1e8-4375-b60b-0698813371eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bec797742cb75a79d4750cd29ec7a8037baf43a3447d2686de3960ad066a4ac5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25096, \"end_char_idx\": 26794, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "\"}],\n    deployment_id=\"gpt-4\",\n    dataSources=[\n        {\n            \"type\": \"AzureCognitiveSearch\",\n            \"parameters\": {\n                \"endpoint\": os.environ[\"SEARCH_ENDPOINT\"],\n                \"key\": os.environ[\"SEARCH_KEY\"],\n                \"indexName\": os.environ[\"SEARCH_INDEX_NAME\"],\n            }\n        }\n    ],\n    stream=True,\n)\n\nfor chunk in response:\n    delta = chunk.choices[0].delta\n\n    if \"role\" in delta:\n        print(\"\\n\"+ delta.role + \": \", end=\"\", flush=True)\n    if \"content\" in delta:\n        print(delta.content, end=\"\", flush=True)\n    if \"context\" in delta:\n        print(f\"Context: {delta.context}\", end=\"\", flush=True)\n```\n\n    Context: {\n      \"messages\": [\n        {\n          \"role\": \"tool\",\n          \"content\": \"{\\\"citations\\\":[{\\\"content\\\":\\\"<h2 id=\\\\\\\"how-are-azure-ai-services-and-azure-machine-learning-aml-similar\\\\\\\">How are Azure AI services and Azure Machine Learning (AML) similar?.</h2>\\\\n<p>Both have the end-goal of applying artificial intelligence (AI) to enhance business operations, though how each provides this in the respective offerings is different..</p>\\\\n<p>Generally, the audiences are different:</p>\\\\n<ul>\\\\n<li>Azure AI services are for developers without machine-learning experience..</li>\\\\n<li>Azure Machine Learning is tailored for data scientists.\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 63eb6346-a1e8-4375-b60b-0698813371eb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"63eb6346-a1e8-4375-b60b-0698813371eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"676909cb-6f06-455d-8edb-298195192d22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a9b41c569e32af5df5fa513ec7617c9870a7a696379292ed7018ebe84250b25e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22a605c5-9fc8-4e27-a2be-819b37a68d6d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2997ab36c5192865ad0730b8e6b6302c062324ee8c8dd511a6034c7ff7948e41\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25096, \"end_char_idx\": 29091, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "\"}],\n    deployment_id=\"gpt-4\",\n    dataSources=[\n        {\n            \"type\": \"AzureCognitiveSearch\",\n            \"parameters\": {\n                \"endpoint\": os.environ[\"SEARCH_ENDPOINT\"],\n                \"key\": os.environ[\"SEARCH_KEY\"],\n                \"indexName\": os.environ[\"SEARCH_INDEX_NAME\"],\n            }\n        }\n    ],\n    stream=True,\n)\n\nfor chunk in response:\n    delta = chunk.choices[0].delta\n\n    if \"role\" in delta:\n        print(\"\\n\"+ delta.role + \": \", end=\"\", flush=True)\n    if \"content\" in delta:\n        print(delta.content, end=\"\", flush=True)\n    if \"context\" in delta:\n        print(f\"Context: {delta.context}\", end=\"\", flush=True)\n```\n\n    Context: {\n      \"messages\": [\n        {\n          \"role\": \"tool\",\n          \"content\": \"{\\\"citations\\\":[{\\\"content\\\":\\\"<h2 id=\\\\\\\"how-are-azure-ai-services-and-azure-machine-learning-aml-similar\\\\\\\">How are Azure AI services and Azure Machine Learning (AML) similar?.</h2>\\\\n<p>Both have the end-goal of applying artificial intelligence (AI) to enhance business operations, though how each provides this in the respective offerings is different..</p>\\\\n<p>Generally, the audiences are different:</p>\\\\n<ul>\\\\n<li>Azure AI services are for developers without machine-learning experience..</li>\\\\n<li>Azure Machine Learning is tailored for data scientists.\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 2/Highlights=30 of size=137\\\"},\\\"chunk_id\\\":\\\"2\\\"},{\\\"content\\\":\\\"<td>Convert speech into text and text into natural-sounding speech..Translate from one language to another and enable speaker verification and recognition..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/vision/\\\\\\\">Vision</a></td>\\\\n<td>Recognize, identify, caption, index, and moderate your pictures, videos, and digital ink content..</td>\\\\n</tr>\\\\n<p></tbody>\\\\n</table></p>\\\\n<p>Use Azure AI services when you:</p>\\\\n<ul>\\\\n<li>Can use a generalized solution..</li>\\\\n</ul>\\\\n<p>Use other machine-learning solutions when you:</p>\\\\n<ul>\\\\n<li>Need to choose the algorithm and need to train on very specific data..</li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"what-is-machine-learning\\\\\\\">What is machine learning?.</h2>\\\\n<p>Machine learning is a concept where you bring together data and an algorithm to solve a specific need..Once the data and algorithm are trained, the output is a model that you can use again with different data..The trained model provides insights based on the new data..</p>\\\\n<p>The process of building a machine learning system requires some knowledge of machine learning or data science..</p>\\\\n<p>Machine learning is provided using <a href=\\\\\\\"/azure/architecture/data-guide/technology-choices/data-science-and-machine-learning?.context=azure%2fmachine-learning%2fstudio%2fcontext%2fml-context\\\\\\\">Azure Machine Learning (AML) products and services</a>..</p>\\\\n<h2 id=\\\\\\\"what-is-an-azure-ai-service\\\\\\\">What is an Azure AI service?.</h2>\\\\n<p>An Azure AI service provides part or all of the components in a machine learning solution: data, algorithm, and trained model..These services are meant to require general knowledge about your data without needing experience with machine learning or data science..These services provide both REST API(s) and language-based SDKs..As a result, you need to have programming language knowledge to use the services..</p>\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1188."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 22a605c5-9fc8-4e27-a2be-819b37a68d6d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"22a605c5-9fc8-4e27-a2be-819b37a68d6d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63eb6346-a1e8-4375-b60b-0698813371eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b52a71ef19077e3c7603876d21cfaead6f2b0d94b71f34b7d6c9c40f9223ebdd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d242ebae-2e78-4e4c-b1cb-dc3536abc668\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7de260f6d4ca30c8bfb0319281dc8fa16250ea079e43db30af7ed6480cba6a43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29092, \"end_char_idx\": 32110, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 1/Highlights=67 of size=506\\\"},\\\"chunk_id\\\":\\\"1\\\"},{\\\"content\\\":\\\"<hr />\\\\n<p>title: Azure AI services and Machine Learning\\\\ntitleSuffix: Azure AI services\\\\ndescription: Learn where Azure AI services fits in with other Azure offerings for machine learning.\\\\nservices: cognitive-services\\\\nmanager: nitinme\\\\nauthor: aahill\\\\nms.author: aahi\\\\nms.service: cognitive-services\\\\nms.topic: conceptual\\\\nms.date: 10/28/2021</p>\\\\n<hr />\\\\n<h1 id=\\\\\\\"azure-ai-services-and-machine-learning\\\\\\\">Azure AI services and machine learning</h1>\\\\n<p>Azure AI services provides machine learning capabilities to solve general problems such as analyzing text for emotional sentiment or analyzing images to recognize objects or faces..You don't need special machine learning or data science knowledge to use these services../what-are-ai-services.md\\\\\\\">Azure AI services</a> is a group of services, each supporting different, generalized prediction capabilities..The services are divided into different categories to help you find the right service..</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Service category</th>\\\\n<th>Purpose</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/decision/\\\\\\\">Decision</a></td>\\\\n<td>Build apps that surface recommendations for informed and efficient decision-making..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/lang/\\\\\\\">Language</a></td>\\\\n<td>Allow your apps to process natural language with pre-built scripts, evaluate sentiment and learn how to recognize what users want..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/search/\\\\\\\">Search</a></td>\\\\n<td>Add Bing Search APIs to your apps and harness the ability to comb billions of webpages, images, videos, and news with a single API call..</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"https://azure.microsoft.com/services/cognitive-services/directory/speech/\\\\\\\">Speech</a></td>\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1188. Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 0/Highlights=63 of size=526\\\"},\\\"chunk_id\\\":\\\"0\\\"},{\\\"content\\\":\\\"<p>How is Azure Cognitive Search related to Azure AI services?</p>\\\\n<p><a href=\\\\\\\"../search/search-what-is-azure-search.md\\\\\\\">Azure Cognitive Search</a> is a separate cloud search service that optionally uses Azure AI services to add image and natural language processing to indexing workloads. Azure AI services is exposed in Azure Cognitive Search through <a href=\\\\\\\"../search/cognitive-search-predefined-skills.md\\\\\\\">built-in skills</a> that wrap individual APIs."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d242ebae-2e78-4e4c-b1cb-dc3536abc668
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d242ebae-2e78-4e4c-b1cb-dc3536abc668\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"22a605c5-9fc8-4e27-a2be-819b37a68d6d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"58888cacfcdca0cc08e80f54b672cc35d2daf1b695bc90f3fd3d4ed4b737cd4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adbbfa90-8f5c-48b7-9752-321c88028f25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0c0bed4ea213b8c8400bf976b61ab85104fbec0dd50ac4804fc9a3988b7b957c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31501, \"end_char_idx\": 34249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "Scores=5.689296 and None.Org Highlight count=160.Filtering to chunk no. 0/Highlights=63 of size=526\\\"},\\\"chunk_id\\\":\\\"0\\\"},{\\\"content\\\":\\\"<p>How is Azure Cognitive Search related to Azure AI services?</p>\\\\n<p><a href=\\\\\\\"../search/search-what-is-azure-search.md\\\\\\\">Azure Cognitive Search</a> is a separate cloud search service that optionally uses Azure AI services to add image and natural language processing to indexing workloads. Azure AI services is exposed in Azure Cognitive Search through <a href=\\\\\\\"../search/cognitive-search-predefined-skills.md\\\\\\\">built-in skills</a> that wrap individual APIs. You can use a free resource for walkthroughs, but plan on creating and attaching a <a href=\\\\\\\"../search/cognitive-search-attach-cognitive-services.md\\\\\\\">billable resource</a> for larger volumes.</p>\\\\n<h2 id=\\\\\\\"how-can-you-use-azure-ai-services\\\\\\\">How can you use Azure AI services?</h2>\\\\n<p>Each service provides information about your data. You can combine services together to chain solutions such as converting speech (audio) to text, translating the text into many languages, then using the translated languages to get answers from a knowledge base. While Azure AI services can be used to create intelligent solutions on their own, they can also be combined with traditional machine learning projects to supplement models or accelerate the development process. </p>\\\\n<p>Azure AI services that provide exported models for other machine learning tools:</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>Model information</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./custom-vision-service/overview.md\\\\\\\">Custom Vision</a></td>\\\\n<td><a href=\\\\\\\"./custom-vision-service/export-model-python.md\\\\\\\">Export</a> for Tensorflow for Android, CoreML for iOS11, ONNX for Windows ML</td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<h2 id=\\\\\\\"learn-more\\\\\\\">Learn more</h2>\\\\n<ul>\\\\n<li><a href=\\\\\\\"/azure/architecture/data-guide/technology-choices/data-science-and-machine-learning\\\\\\\">Architecture Guide - What are the machine learning products at Microsoft?</a></li>\\\\n<li><a href=\\\\\\\"../machine-learning/concept-deep-learning-vs-machine-learning.md\\\\\\\">Machine learning - Introduction to deep learning vs. machine learning</a></li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"next-steps\\\\\\\">Next steps</h2>\\\\n<ul>\\\\n<li>Create your Azure AI services resource in the <a href=\\\\\\\"multi-service-resource.md?pivots=azportal\\\\\\\">Azure portal</a> or with <a href=\\\\\\\"./multi-service-resource.md?pivots=azcli\\\\\\\">Azure CLI</a>.</li>\\\\n<li>Learn how to <a href=\\\\\\\"authentication.md\\\\\\\">authenticate</a> with your Azure AI service.</li>\\\\n<li>Use <a href=\\\\\\\"diagnostic-logging.md\\\\\\\">diagnostic logging</a> for issue identification and debugging."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: adbbfa90-8f5c-48b7-9752-321c88028f25
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"adbbfa90-8f5c-48b7-9752-321c88028f25\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d242ebae-2e78-4e4c-b1cb-dc3536abc668\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"339af496b7f21702599435ea68dc99f3d56e4804757c442ad131f42aa7c78287\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3e6a5d90-ca25-4373-a031-38fba6f2674e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"940890d952c47ce6d0e48b62dc26ee5212972543e8268b96044848cd079b2fa8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "machine learning</a></li>\\\\n</ul>\\\\n<h2 id=\\\\\\\"next-steps\\\\\\\">Next steps</h2>\\\\n<ul>\\\\n<li>Create your Azure AI services resource in the <a href=\\\\\\\"multi-service-resource.md?pivots=azportal\\\\\\\">Azure portal</a> or with <a href=\\\\\\\"./multi-service-resource.md?pivots=azcli\\\\\\\">Azure CLI</a>.</li>\\\\n<li>Learn how to <a href=\\\\\\\"authentication.md\\\\\\\">authenticate</a> with your Azure AI service.</li>\\\\n<li>Use <a href=\\\\\\\"diagnostic-logging.md\\\\\\\">diagnostic logging</a> for issue identification and debugging. </li>\\\\n<li>Deploy an Azure AI service in a Docker <a href=\\\\\\\"cognitive-services-container-support.md\\\\\\\">container</a>.</li>\\\\n<li>Keep up to date with <a href=\\\\\\\"https://azure.microsoft.com/updates/?product=cognitive-services\\\\\\\">service updates</a>.</li>\\\\n</ul>\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=793. Scores=3.3767838 and None.Org Highlight count=69.\\\"},\\\"chunk_id\\\":\\\"3\\\"},{\\\"content\\\":\\\"<p>How are Azure AI services different from machine learning?.</p>\\\\n<p>Azure AI services provide a trained model for you.This brings data and an algorithm together, available from a REST API(s) or SDK.An Azure AI service provides answers to general problems such as key phrases in text or item identification in images.</p>\\\\n<p>Machine learning is a process that generally requires a longer period of time to implement successfully.This time is spent on data collection, cleaning, transformation, algorithm selection, model training, and deployment to get to the same level of functionality provided by an Azure AI service.With machine learning, it is possible to provide answers to highly specialized and/or specific problems.Machine learning problems require familiarity with the specific subject matter and data of the problem under consideration, as well as expertise in data science.</p>\\\\n<h2 id=\\\\\\\"what-kind-of-data-do-you-have\\\\\\\">What kind of data do you have?.</h2>\\\\n<p>Azure AI services, as a group of services, can require none, some, or all custom data for the trained model.</p>\\\\n<h3 id=\\\\\\\"no-additional-training-data-required\\\\\\\">No additional training data required</h3>\\\\n<p>Services that provide a fully-trained model can be treated as a <em>opaque box</em>.You don't need to know how they work or what data was used to train them.</p>\\\\n<h3 id=\\\\\\\"some-or-all-training-data-required\\\\\\\">Some or all training data required</h3>\\\\n<p>Some services allow you to bring your own data, then train a model.This allows you to extend the model using the Service's data and algorithm with your own data.The output matches your needs.When you bring your own data, you may need to tag the data in a way specific to the service.For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model.These services process significant amounts of model data."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3e6a5d90-ca25-4373-a031-38fba6f2674e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3e6a5d90-ca25-4373-a031-38fba6f2674e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adbbfa90-8f5c-48b7-9752-321c88028f25\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bd17b314877e9cdab305a11bbe5e5ec75e78cc76d1536986e5b364c79ab21713\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"499c6b84-a905-4699-8422-b3de18989e3c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5ff41f842687bceef9d4ad461c29f6f4371a67c506b5cbaa530a3c88c8fb013b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "An Azure AI service provides answers to general problems such as key phrases in text or item identification in images.</p>\\\\n<p>Machine learning is a process that generally requires a longer period of time to implement successfully.This time is spent on data collection, cleaning, transformation, algorithm selection, model training, and deployment to get to the same level of functionality provided by an Azure AI service.With machine learning, it is possible to provide answers to highly specialized and/or specific problems.Machine learning problems require familiarity with the specific subject matter and data of the problem under consideration, as well as expertise in data science.</p>\\\\n<h2 id=\\\\\\\"what-kind-of-data-do-you-have\\\\\\\">What kind of data do you have?.</h2>\\\\n<p>Azure AI services, as a group of services, can require none, some, or all custom data for the trained model.</p>\\\\n<h3 id=\\\\\\\"no-additional-training-data-required\\\\\\\">No additional training data required</h3>\\\\n<p>Services that provide a fully-trained model can be treated as a <em>opaque box</em>.You don't need to know how they work or what data was used to train them.</p>\\\\n<h3 id=\\\\\\\"some-or-all-training-data-required\\\\\\\">Some or all training data required</h3>\\\\n<p>Some services allow you to bring your own data, then train a model.This allows you to extend the model using the Service's data and algorithm with your own data.The output matches your needs.When you bring your own data, you may need to tag the data in a way specific to the service.For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model.These services process significant amounts of model data.</p>\\\\n<h2 id=\\\\\\\"service-requirements-for-the-data-model\\\\\\\">Service requirements for the data model</h2>\\\\n<p>The following data categorizes each service by which kind of data it allows or requires.</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>No training data required</th>\\\\n<th>You provide some or all training data</th>\\\\n<th>Real-time or near real-time data collection</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./LUIS/what-is-luis.md\\\\\\\">Language Understanding (LUIS)</a></td>\\\\n<td></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./personalizer/what-is-personalizer.md\\\\\\\">Personalizer</a><sup>1</sup></sup></td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./computer-vision/overview.md\\\\\\\">Vision</a></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n<td></td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<p><sup>1</sup> Personalizer only needs training data collected by the service (as it operates in real-time) to evaluate your policy and data."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 499c6b84-a905-4699-8422-b3de18989e3c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"499c6b84-a905-4699-8422-b3de18989e3c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3e6a5d90-ca25-4373-a031-38fba6f2674e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f7d2955fa26a821454118cf2604fc15c4f5d9f9377372faa864dea98b17b614d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9baf1e94-bfc3-4d8e-b4b0-3da1f41493ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3202515416a0079d3f930ddc20a4afa85dc352479a0af5454ed00aa5b16d4214\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "For example, if you are training a model to identify flowers, you can provide a catalog of flower images along with the location of the flower in each image to train the model.These services process significant amounts of model data.</p>\\\\n<h2 id=\\\\\\\"service-requirements-for-the-data-model\\\\\\\">Service requirements for the data model</h2>\\\\n<p>The following data categorizes each service by which kind of data it allows or requires.</p>\\\\n<table>\\\\n<thead>\\\\n<tr>\\\\n<th>Azure AI service</th>\\\\n<th>No training data required</th>\\\\n<th>You provide some or all training data</th>\\\\n<th>Real-time or near real-time data collection</th>\\\\n</tr>\\\\n</thead>\\\\n<tbody>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./LUIS/what-is-luis.md\\\\\\\">Language Understanding (LUIS)</a></td>\\\\n<td></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./personalizer/what-is-personalizer.md\\\\\\\">Personalizer</a><sup>1</sup></sup></td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n<td>x</td>\\\\n</tr>\\\\n<tr>\\\\n<td><a href=\\\\\\\"./computer-vision/overview.md\\\\\\\">Vision</a></td>\\\\n<td>x</td>\\\\n<td></td>\\\\n<td></td>\\\\n</tr>\\\\n</tbody>\\\\n</table>\\\\n<p><sup>1</sup> Personalizer only needs training data collected by the service (as it operates in real-time) to evaluate your policy and data.</p>\\\\n<h2 id=\\\\\\\"where-can-you-use-azure-ai-services\\\\\\\">Where can you use Azure AI services?.</h2>\\\\n<p>The services are used in any application that can make REST API(s) or SDK calls.Examples of applications include web sites, bots, virtual or mixed reality, desktop and mobile applications.\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1734. Scores=3.1447978 and None.Org Highlight count=66.Filtering to highlight size=891\\\"},\\\"chunk_id\\\":\\\"4\\\"}],\\\"intent\\\":\\\"[\\\\\\\"What are the differences between Azure Machine Learning and Azure AI services?\\\\\\\"]\\\"}\",\n          \"end_turn\": false\n        }\n      ]\n    }\n    assistant: Azure AI services and Azure Machine Learning (AML) both aim to apply artificial intelligence (AI) to enhance business operations, but they target different audiences and offer different capabilities [doc1]. \n    \n    Azure AI services are designed for developers without machine learning experience and provide pre-trained models to solve general problems such as text analysis, image recognition, and natural language processing [doc5]. These services require general knowledge about your data without needing experience with machine learning or data science and provide REST APIs and language-based SDKs [doc2].\n    \n    On the other hand, Azure Machine Learning is tailored for data scientists and offers a platform to build, train, and deploy custom machine learning models [doc1]."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9baf1e94-bfc3-4d8e-b4b0-3da1f41493ec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb", "title": "Azure chat completion models with your own data (preview)", "tokens": 9636, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9baf1e94-bfc3-4d8e-b4b0-3da1f41493ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8a416323-9970-4bd9-8494-b6e70dc26de3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45efe248acff036fd32ca8bf274a616f2bb949b916b042f42ce70ce3d9a972f0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"499c6b84-a905-4699-8422-b3de18989e3c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/azure/archive/chat_with_your_own_data.ipynb\", \"title\": \"Azure chat completion models with your own data (preview)\", \"tokens\": 9636, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c9f2645ece21e96e3d47a8d9c9f0ce0861a588f282ba6319e9ccd1eef661890a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "ref_doc_id": "8a416323-9970-4bd9-8494-b6e70dc26de3", "chroma:document": "</p>\\\\n<h2 id=\\\\\\\"where-can-you-use-azure-ai-services\\\\\\\">Where can you use Azure AI services?.</h2>\\\\n<p>The services are used in any application that can make REST API(s) or SDK calls.Examples of applications include web sites, bots, virtual or mixed reality, desktop and mobile applications.\\\",\\\"id\\\":null,\\\"title\\\":\\\"Azure AI services and machine learning\\\",\\\"filepath\\\":\\\"cognitive-services-and-machine-learning.md\\\",\\\"url\\\":\\\"https://krpraticstorageacc.blob.core.windows.net/azure-openai/cognitive-services-and-machine-learning.md\\\",\\\"metadata\\\":{\\\"chunking\\\":\\\"orignal document size=1734. Scores=3.1447978 and None.Org Highlight count=66.Filtering to highlight size=891\\\"},\\\"chunk_id\\\":\\\"4\\\"}],\\\"intent\\\":\\\"[\\\\\\\"What are the differences between Azure Machine Learning and Azure AI services?\\\\\\\"]\\\"}\",\n          \"end_turn\": false\n        }\n      ]\n    }\n    assistant: Azure AI services and Azure Machine Learning (AML) both aim to apply artificial intelligence (AI) to enhance business operations, but they target different audiences and offer different capabilities [doc1]. \n    \n    Azure AI services are designed for developers without machine learning experience and provide pre-trained models to solve general problems such as text analysis, image recognition, and natural language processing [doc5]. These services require general knowledge about your data without needing experience with machine learning or data science and provide REST APIs and language-based SDKs [doc2].\n    \n    On the other hand, Azure Machine Learning is tailored for data scientists and offers a platform to build, train, and deploy custom machine learning models [doc1]. It requires knowledge of machine learning or data science and allows users to choose the algorithm and train on very specific data [doc2].\n    \n    In summary, Azure AI services offer pre-trained models for developers without machine learning experience, while Azure Machine Learning is a platform for data scientists to build and deploy custom machine learning models."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: fc655a19-70b9-4d15-b35a-e83b79d5ad88
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"fc655a19-70b9-4d15-b35a-e83b79d5ad88\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"87496997-7b1a-4ff3-942d-07efc55394a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59b13ccba574a65fddad89c43e9ee813a49c482fc36db59b66e7352709d90877\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2986, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "# Translate a book written in LaTeX from Slovenian into English\n\nWith permission of the author, we will demonstrate how to translate the book [Euclidean Plane Geometry](https://sites.google.com/site/projektivna/), written by Milan Mitrovi\u0107 from Slovenian into English, without modifying any of the LaTeX commands.\n\nTo achieve this, we will first split the book into chunks, each roughly a page long, then translate each chunk into English, and finally stitch them back together.\n\n## 1. Read in the data\n\n\n```python\nfrom openai import OpenAI\nimport os\nfrom transformers import GPT2Tokenizer\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if you didn't set as an env var>\"))\n\n# OpenAI GPT-2 tokenizer is the same as GPT-3 tokenizer\n# we use it to count the number of tokens in the text\ntokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n\nwith open(\"data/geometry_slovenian.tex\", \"r\") as f:\n    text = f.read()\n```\n\n\n\n\n    1485565\n\n\n\n### 1.1 Count the tokens in each chunk\n\n\n```python\nchunks = text.split('\\n\\n')\nntokens = []\nfor chunk in chunks:\n    ntokens.append(len(tokenizer.encode(chunk)))\nmax(ntokens)\n```\n\n    Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors\n\n\n\n\n\n    1473\n\n\n\nIt turns out that a double newline is a good separator in this case, in order not to break the flow of the text. Also no individual chunk is larger than 1500 tokens. The model we will use is text-davinci-002, which has a limit of 4096 tokens, so we don't need to worry about breaking the chunks down further.\n\nWe will group the shorter chunks into chunks of around 1000 tokens, to increase the coherence of the text, and decrease the frequency of breaks within the text.\n\n\n```python\ndef group_chunks(chunks, ntokens, max_len=1000, hard_max_len=3000):\n    \"\"\"\n    Group very short chunks, to form approximately page long chunks.\n    \"\"\"\n    batches = []\n    cur_batch = \"\"\n    cur_tokens = 0\n    \n    # iterate over chunks, and group the short ones together\n    for chunk, ntoken in zip(chunks, ntokens):\n        # discard chunks that exceed hard max length\n        if ntoken > hard_max_len:\n            print(f\"Warning: Chunk discarded for being too long ({ntoken} tokens > {hard_max_len} token limit). Preview: '{chunk[:50]}...'\")\n            continue\n\n        # if room in current batch, add new chunk\n        if cur_tokens + 1 + ntoken <= max_len:\n            cur_batch += \"\\n\\n\" + chunk\n            cur_tokens += 1 + ntoken  # adds 1 token for the two newlines\n        # otherwise, record the batch and start a new one\n        else:\n            batches.append(cur_batch)\n            cur_batch = chunk\n            cur_tokens = ntoken\n            \n    if cur_batch:  # add the last batch if it's not empty\n        batches.append(cur_batch)\n        \n    return batches\n\n\nchunks = group_chunks(chunks, ntokens)\nlen(chunks)\n```\n\n\n\n\n    869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 87496997-7b1a-4ff3-942d-07efc55394a7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"87496997-7b1a-4ff3-942d-07efc55394a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fc655a19-70b9-4d15-b35a-e83b79d5ad88\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"54d9fbcaecac05b58e73f6fbcf2759b00b45f0afb7829eb239088bb0879b9d19\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63c68992-ad9a-4543-8afd-baba6a7f86fd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b3ed6b41e06d266bd7e812e3757cc9c888fcc18fe5284eca22b8deccf844cac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1340, \"end_char_idx\": 3987, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "1473\n\n\n\nIt turns out that a double newline is a good separator in this case, in order not to break the flow of the text. Also no individual chunk is larger than 1500 tokens. The model we will use is text-davinci-002, which has a limit of 4096 tokens, so we don't need to worry about breaking the chunks down further.\n\nWe will group the shorter chunks into chunks of around 1000 tokens, to increase the coherence of the text, and decrease the frequency of breaks within the text.\n\n\n```python\ndef group_chunks(chunks, ntokens, max_len=1000, hard_max_len=3000):\n    \"\"\"\n    Group very short chunks, to form approximately page long chunks.\n    \"\"\"\n    batches = []\n    cur_batch = \"\"\n    cur_tokens = 0\n    \n    # iterate over chunks, and group the short ones together\n    for chunk, ntoken in zip(chunks, ntokens):\n        # discard chunks that exceed hard max length\n        if ntoken > hard_max_len:\n            print(f\"Warning: Chunk discarded for being too long ({ntoken} tokens > {hard_max_len} token limit). Preview: '{chunk[:50]}...'\")\n            continue\n\n        # if room in current batch, add new chunk\n        if cur_tokens + 1 + ntoken <= max_len:\n            cur_batch += \"\\n\\n\" + chunk\n            cur_tokens += 1 + ntoken  # adds 1 token for the two newlines\n        # otherwise, record the batch and start a new one\n        else:\n            batches.append(cur_batch)\n            cur_batch = chunk\n            cur_tokens = ntoken\n            \n    if cur_batch:  # add the last batch if it's not empty\n        batches.append(cur_batch)\n        \n    return batches\n\n\nchunks = group_chunks(chunks, ntokens)\nlen(chunks)\n```\n\n\n\n\n    869\n\n\n\nNotice that adding a sample untranslated and translated first command, where only the content of the chapter name needs to be translated, helps to get more consistent results.\n\nThe format of the prompt sent to the model consists of:\n1. A high level instruction to translate only the text, but not commands into the desired language\n2. A sample untranslated command, where only the content of the chapter name needs to be translated\n3. The chunk of text to be translated\n4. The translated sample command from 2, which shows the model the beginning of the translation process\n\nThe expected output is the translated chunk of text.\n\n\n```python\ndef translate_chunk(chunk, model='gpt-3.5-turbo',\n                    dest_language='English',\n                    sample_translation=(\"\\poglavje{Osnove Geometrije} \\label{osn9Geom}\", \"\\poglavje{The basics of Geometry} \\label{osn9Geom}\")\n                    ):\n    prompt = f'''Translate only the text from the following LaTeX document into {dest_language}."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 63c68992-ad9a-4543-8afd-baba6a7f86fd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"63c68992-ad9a-4543-8afd-baba6a7f86fd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"87496997-7b1a-4ff3-942d-07efc55394a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2ebae210bb4c865a0010fffab3752dbb3ec7aeb8a61e7a47209f63838c9a8e75\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3dabe3a7-fe89-4094-ad4b-307b2b12f0b4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a782b2ad4c11a9f3d576071942f9dd0e0ff9b280b6d96d7e1d4d340c38a87c98\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2920, \"end_char_idx\": 5561, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "chunks = group_chunks(chunks, ntokens)\nlen(chunks)\n```\n\n\n\n\n    869\n\n\n\nNotice that adding a sample untranslated and translated first command, where only the content of the chapter name needs to be translated, helps to get more consistent results.\n\nThe format of the prompt sent to the model consists of:\n1. A high level instruction to translate only the text, but not commands into the desired language\n2. A sample untranslated command, where only the content of the chapter name needs to be translated\n3. The chunk of text to be translated\n4. The translated sample command from 2, which shows the model the beginning of the translation process\n\nThe expected output is the translated chunk of text.\n\n\n```python\ndef translate_chunk(chunk, model='gpt-3.5-turbo',\n                    dest_language='English',\n                    sample_translation=(\"\\poglavje{Osnove Geometrije} \\label{osn9Geom}\", \"\\poglavje{The basics of Geometry} \\label{osn9Geom}\")\n                    ):\n    prompt = f'''Translate only the text from the following LaTeX document into {dest_language}. Leave all LaTeX commands unchanged\n    \n\"\"\"\n{sample_translation[0]}\n{chunk}\"\"\"\n\n{sample_translation[1]}\n'''\n    response = client.chat.completions.create(\n        messages=[{\"role\": \"user\", \"content\":prompt}],\n        model=model,\n        temperature=0,\n        top_p=1,\n        max_tokens=1500,\n    )\n    result = response.choices[0].message.content.strip()\n    result = result.replace('\"\"\"', '') # remove the double quotes, as we used them to surround the text\n    return result\nprint(translate_chunk(chunks[800], model='gpt-3.5-turbo', dest_language='English'))\n```\n\n    Let $\\mathcal{I}=\\mathcal{S}_{AB} \\circ\\mathcal{S}_{CA}\n        \\circ\\mathcal{S}_{BC}$. By  \\ref{izoZrcdrsprq} is\n        $\\mathcal{I}$ a mirror reflection. Let $A_1$, $B_1$ and $C_1$ be in order the center points of the lines $BC$, $AC$ and $AB$ of the triangle $ABC$.\n        Because it is a right triangle is $\\mathcal{I}(A_1C_1)=A_1C_1$, which\n        means that the line $A_1C_1$ is of this mirror reflection. It is not\n        difficult to prove that for the point $A'_1=\\mathcal{I}(A_1)$ (both\n        lie on the axis $A_1C_1$) is\n        $\\overrightarrow{A_1A'_1}=3\\overrightarrow{A_1C_1}$, so\n        $\\mathcal{I}=\\mathcal{G}_{3\\overrightarrow{A_1C_1}}$.\n    \n    \\item  \\res{Given are the points $A$ and $B$ on the same side of the line\n    $p$.\n    Draw the line  $XY$, which lies on the line $p$ and is consistent\n    with the given line $l$, so that the sum\n    $|AX|+|XY|+|YB|$ is minimal.}\n    \n    Let $A'=\\mathcal{G}_{\\overrightarrow{MN}}(A)$ (where $M,N\\in\n    p$ and $MN\\cong l$)."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3dabe3a7-fe89-4094-ad4b-307b2b12f0b4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3dabe3a7-fe89-4094-ad4b-307b2b12f0b4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63c68992-ad9a-4543-8afd-baba6a7f86fd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6fe5ceb09c3629d441c56b9652330ab692a51f3e3e2662b2220818ebd07992fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7dbb06af-48a3-4347-a5e5-ebb5366ccefb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2da577e8eb02dc10820083d09fc69187f372063153f894a604e6d30ca466bd8a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4650, \"end_char_idx\": 6713, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "By  \\ref{izoZrcdrsprq} is\n        $\\mathcal{I}$ a mirror reflection. Let $A_1$, $B_1$ and $C_1$ be in order the center points of the lines $BC$, $AC$ and $AB$ of the triangle $ABC$.\n        Because it is a right triangle is $\\mathcal{I}(A_1C_1)=A_1C_1$, which\n        means that the line $A_1C_1$ is of this mirror reflection. It is not\n        difficult to prove that for the point $A'_1=\\mathcal{I}(A_1)$ (both\n        lie on the axis $A_1C_1$) is\n        $\\overrightarrow{A_1A'_1}=3\\overrightarrow{A_1C_1}$, so\n        $\\mathcal{I}=\\mathcal{G}_{3\\overrightarrow{A_1C_1}}$.\n    \n    \\item  \\res{Given are the points $A$ and $B$ on the same side of the line\n    $p$.\n    Draw the line  $XY$, which lies on the line $p$ and is consistent\n    with the given line $l$, so that the sum\n    $|AX|+|XY|+|YB|$ is minimal.}\n    \n    Let $A'=\\mathcal{G}_{\\overrightarrow{MN}}(A)$ (where $M,N\\in\n    p$ and $MN\\cong l$). The point $Y$ is obtained as the intersection of the lines $p$\n    and $X'Y$ (see also example \\ref{HeronProbl}).\n    \n    \\item  \\res{Let $ABC$ be an isosceles right triangle with a right angle at the vertex $A$. What does the composite\n    $\\mathcal{G}_{\\overrightarrow{AB}}\\circ \\mathcal{G}_{\\overrightarrow{CA}}$ represent?}\n    \n    Let $p$ and $q$ be the simetrali of the sides $CA$ and $AB$ of the triangle\n    $ABC$. By  \\ref{izoZrcDrsKompSrOsn} is:\n     $$\\mathcal{G}_{\\overrightarrow{AB}}\\circ\n     \\mathcal{G}_{\\overrightarrow{CA}}=\n     \\mathcal{S}_q\\circ\\mathcal{S}_A\\circ\\mathcal{S}_A\\circ\\mathcal{S}_p=\n     \\mathcal{S}_q\\circ\\mathcal{S}_p.$$ Because $ABC$ is an isosceles\n     right triangle with a right angle at the vertex $A$, the lines $p$ and $q$ are perpendicular and intersect at the center $S$\n     of the hypotenuse $BC$. Therefore\n     $\\mathcal{G}_{\\overrightarrow{AB}}\\circ\n     \\mathcal{G}_{\\overrightarrow{CA}}=\\mathcal{S}_q\n     \\circ\\mathcal{S}_p=\\mathcal{S}_S$.\n    \n    \\item \\res{In the same plane are given the lines\n    $a$, $b$ and $c$.\n    Draw the points $A\\in a$ and $B\\in b$\n    so that $\\mathcal{S}_c(A)=B$.}"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7dbb06af-48a3-4347-a5e5-ebb5366ccefb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7dbb06af-48a3-4347-a5e5-ebb5366ccefb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3dabe3a7-fe89-4094-ad4b-307b2b12f0b4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bd72d5f4cf20a46191edb0259a94623cde612b11ca94e49c7648d65181ab4700\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c29b2d3a-926a-4280-879c-0dc9318a5943\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b16408b18cdb67d6c3acaa8c9af161f4b854c4e53fe8ad745d0096ee85e6b151\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5685, \"end_char_idx\": 7712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "\\item  \\res{Let $ABC$ be an isosceles right triangle with a right angle at the vertex $A$. What does the composite\n    $\\mathcal{G}_{\\overrightarrow{AB}}\\circ \\mathcal{G}_{\\overrightarrow{CA}}$ represent?}\n    \n    Let $p$ and $q$ be the simetrali of the sides $CA$ and $AB$ of the triangle\n    $ABC$. By  \\ref{izoZrcDrsKompSrOsn} is:\n     $$\\mathcal{G}_{\\overrightarrow{AB}}\\circ\n     \\mathcal{G}_{\\overrightarrow{CA}}=\n     \\mathcal{S}_q\\circ\\mathcal{S}_A\\circ\\mathcal{S}_A\\circ\\mathcal{S}_p=\n     \\mathcal{S}_q\\circ\\mathcal{S}_p.$$ Because $ABC$ is an isosceles\n     right triangle with a right angle at the vertex $A$, the lines $p$ and $q$ are perpendicular and intersect at the center $S$\n     of the hypotenuse $BC$. Therefore\n     $\\mathcal{G}_{\\overrightarrow{AB}}\\circ\n     \\mathcal{G}_{\\overrightarrow{CA}}=\\mathcal{S}_q\n     \\circ\\mathcal{S}_p=\\mathcal{S}_S$.\n    \n    \\item \\res{In the same plane are given the lines\n    $a$, $b$ and $c$.\n    Draw the points $A\\in a$ and $B\\in b$\n    so that $\\mathcal{S}_c(A)=B$.}\n\n\nWe can see here that this one chunk in particular translates only the text, but leaves LaTeX commands intact.\n\nLet's now translate all the chunks in the book - this will take 2-3 hours, as we're processing requests sequentially.\n\n\n```python\ndest_language = \"English\"\n\ntranslated_chunks = []\nfor i, chunk in enumerate(chunks):\n    print(str(i+1) + \" / \" + str(len(chunks)))\n    # translate each chunk\n    translated_chunks.append(translate_chunk(chunk, model='gpt-3.5-turbo', dest_language=dest_language))\n\n# join the chunks together\nresult = '\\n\\n'.join(translated_chunks)\n\n# save the final result\nwith open(f\"data/geometry_{dest_language}.tex\", \"w\") as f:\n    f.write(result)\n```\n\n    0 / 869\n    1 / 869\n    2 / 869\n    3 / 869\n    4 / 869\n    5 / 869\n    6 / 869\n    7 / 869\n    8 / 869\n    9 / 869\n    10 / 869\n    11 / 869\n    12 / 869\n    13 / 869\n    14 / 869\n    15 / 869\n    16 / 869\n    17 / 869\n    18 / 869\n    19 / 869\n    20 / 869\n    21 / 869\n    22 / 869\n    23 / 869\n    24 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c29b2d3a-926a-4280-879c-0dc9318a5943
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c29b2d3a-926a-4280-879c-0dc9318a5943\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7dbb06af-48a3-4347-a5e5-ebb5366ccefb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0cec2a5ccddf4b7631187c1972abc99ff1570142fee847acc670de21a655012a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81b3f483-c225-48b1-90c2-2f3ebeb511aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32968ffc0221049c9f4e367a9c884807c98e401297191d73c9752e6460221452\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6716, \"end_char_idx\": 8254, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "We can see here that this one chunk in particular translates only the text, but leaves LaTeX commands intact.\n\nLet's now translate all the chunks in the book - this will take 2-3 hours, as we're processing requests sequentially.\n\n\n```python\ndest_language = \"English\"\n\ntranslated_chunks = []\nfor i, chunk in enumerate(chunks):\n    print(str(i+1) + \" / \" + str(len(chunks)))\n    # translate each chunk\n    translated_chunks.append(translate_chunk(chunk, model='gpt-3.5-turbo', dest_language=dest_language))\n\n# join the chunks together\nresult = '\\n\\n'.join(translated_chunks)\n\n# save the final result\nwith open(f\"data/geometry_{dest_language}.tex\", \"w\") as f:\n    f.write(result)\n```\n\n    0 / 869\n    1 / 869\n    2 / 869\n    3 / 869\n    4 / 869\n    5 / 869\n    6 / 869\n    7 / 869\n    8 / 869\n    9 / 869\n    10 / 869\n    11 / 869\n    12 / 869\n    13 / 869\n    14 / 869\n    15 / 869\n    16 / 869\n    17 / 869\n    18 / 869\n    19 / 869\n    20 / 869\n    21 / 869\n    22 / 869\n    23 / 869\n    24 / 869\n    25 / 869\n    26 / 869\n    27 / 869\n    28 / 869\n    29 / 869\n    30 / 869\n    31 / 869\n    32 / 869\n    33 / 869\n    34 / 869\n    35 / 869\n    36 / 869\n    37 / 869\n    38 / 869\n    39 / 869\n    40 / 869\n    41 / 869\n    42 / 869\n    43 / 869\n    44 / 869\n    45 / 869\n    46 / 869\n    47 / 869\n    48 / 869\n    49 / 869\n    50 / 869\n    51 / 869\n    52 / 869\n    53 / 869\n    54 / 869\n    55 / 869\n    56 / 869\n    57 / 869\n    58 / 869\n    59 / 869\n    60 / 869\n    61 / 869\n    62 / 869\n    63 / 869\n    64 / 869\n    65 / 869\n    66 /"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 81b3f483-c225-48b1-90c2-2f3ebeb511aa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"81b3f483-c225-48b1-90c2-2f3ebeb511aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c29b2d3a-926a-4280-879c-0dc9318a5943\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cee132f99b49de5dc8b9ed78d20304af1d0b8be47582bc38696f196e978db1ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d9df1f0c-2bed-49dd-a16c-a0dd02c25d64\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e82aa04c62b0b5257c876b80baffb3a6786686c5d3071f0d97b117acf715a163\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7678, \"end_char_idx\": 8799, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "22 / 869\n    23 / 869\n    24 / 869\n    25 / 869\n    26 / 869\n    27 / 869\n    28 / 869\n    29 / 869\n    30 / 869\n    31 / 869\n    32 / 869\n    33 / 869\n    34 / 869\n    35 / 869\n    36 / 869\n    37 / 869\n    38 / 869\n    39 / 869\n    40 / 869\n    41 / 869\n    42 / 869\n    43 / 869\n    44 / 869\n    45 / 869\n    46 / 869\n    47 / 869\n    48 / 869\n    49 / 869\n    50 / 869\n    51 / 869\n    52 / 869\n    53 / 869\n    54 / 869\n    55 / 869\n    56 / 869\n    57 / 869\n    58 / 869\n    59 / 869\n    60 / 869\n    61 / 869\n    62 / 869\n    63 / 869\n    64 / 869\n    65 / 869\n    66 / 869\n    67 / 869\n    68 / 869\n    69 / 869\n    70 / 869\n    71 / 869\n    72 / 869\n    73 / 869\n    74 / 869\n    75 / 869\n    76 / 869\n    77 / 869\n    78 / 869\n    79 / 869\n    80 / 869\n    81 / 869\n    82 / 869\n    83 / 869\n    84 / 869\n    85 / 869\n    86 / 869\n    87 / 869\n    88 / 869\n    89 / 869\n    90 / 869\n    91 / 869\n    92 / 869\n    93 / 869\n    94 / 869\n    95 / 869\n    96 / 869\n    97 / 869\n    98 / 869\n    99 / 869\n    100 / 869\n    101 / 869\n    102 / 869\n    103 / 869\n    104 / 869\n    105 / 869\n    106 / 869\n    107 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d9df1f0c-2bed-49dd-a16c-a0dd02c25d64
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d9df1f0c-2bed-49dd-a16c-a0dd02c25d64\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81b3f483-c225-48b1-90c2-2f3ebeb511aa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bd899fa217ec89b779a82e64b72d52aa41c224a7857446dac66a57f7682d2744\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4e247c1-22d1-4907-aef1-4732274a2f00\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"590cd9182bc4c9e79a95dc4f5d314ac10eccecaf6b88f6d7f57a86f8a67102f0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8224, \"end_char_idx\": 9387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "64 / 869\n    65 / 869\n    66 / 869\n    67 / 869\n    68 / 869\n    69 / 869\n    70 / 869\n    71 / 869\n    72 / 869\n    73 / 869\n    74 / 869\n    75 / 869\n    76 / 869\n    77 / 869\n    78 / 869\n    79 / 869\n    80 / 869\n    81 / 869\n    82 / 869\n    83 / 869\n    84 / 869\n    85 / 869\n    86 / 869\n    87 / 869\n    88 / 869\n    89 / 869\n    90 / 869\n    91 / 869\n    92 / 869\n    93 / 869\n    94 / 869\n    95 / 869\n    96 / 869\n    97 / 869\n    98 / 869\n    99 / 869\n    100 / 869\n    101 / 869\n    102 / 869\n    103 / 869\n    104 / 869\n    105 / 869\n    106 / 869\n    107 / 869\n    108 / 869\n    109 / 869\n    110 / 869\n    111 / 869\n    112 / 869\n    113 / 869\n    114 / 869\n    115 / 869\n    116 / 869\n    117 / 869\n    118 / 869\n    119 / 869\n    120 / 869\n    121 / 869\n    122 / 869\n    123 / 869\n    124 / 869\n    125 / 869\n    126 / 869\n    127 / 869\n    128 / 869\n    129 / 869\n    130 / 869\n    131 / 869\n    132 / 869\n    133 / 869\n    134 / 869\n    135 / 869\n    136 / 869\n    137 / 869\n    138 / 869\n    139 / 869\n    140 / 869\n    141 / 869\n    142 / 869\n    143 / 869\n    144 / 869\n    145 / 869\n    146 / 869\n    147 / 869\n    148 / 869\n    149 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e4e247c1-22d1-4907-aef1-4732274a2f00
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e4e247c1-22d1-4907-aef1-4732274a2f00\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d9df1f0c-2bed-49dd-a16c-a0dd02c25d64\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ed5029c5289564d9dbbfd7ab33bb5ed3c37eab0be55ff6ecd2589aff521f36b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"68be47b5-81e0-446b-a38f-f91c8e7ef831\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e8e23f1c0469c9cd462a8e0480d776ba2fccda1b1fd0e9b240e9ecc679e73072\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8766, \"end_char_idx\": 9969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "/ 869\n    106 / 869\n    107 / 869\n    108 / 869\n    109 / 869\n    110 / 869\n    111 / 869\n    112 / 869\n    113 / 869\n    114 / 869\n    115 / 869\n    116 / 869\n    117 / 869\n    118 / 869\n    119 / 869\n    120 / 869\n    121 / 869\n    122 / 869\n    123 / 869\n    124 / 869\n    125 / 869\n    126 / 869\n    127 / 869\n    128 / 869\n    129 / 869\n    130 / 869\n    131 / 869\n    132 / 869\n    133 / 869\n    134 / 869\n    135 / 869\n    136 / 869\n    137 / 869\n    138 / 869\n    139 / 869\n    140 / 869\n    141 / 869\n    142 / 869\n    143 / 869\n    144 / 869\n    145 / 869\n    146 / 869\n    147 / 869\n    148 / 869\n    149 / 869\n    150 / 869\n    151 / 869\n    152 / 869\n    153 / 869\n    154 / 869\n    155 / 869\n    156 / 869\n    157 / 869\n    158 / 869\n    159 / 869\n    160 / 869\n    161 / 869\n    162 / 869\n    163 / 869\n    164 / 869\n    165 / 869\n    166 / 869\n    167 / 869\n    168 / 869\n    169 / 869\n    170 / 869\n    171 / 869\n    172 / 869\n    173 / 869\n    174 / 869\n    175 / 869\n    176 / 869\n    177 / 869\n    178 / 869\n    179 / 869\n    180 / 869\n    181 / 869\n    182 / 869\n    183 / 869\n    184 / 869\n    185 / 869\n    186 / 869\n    187 / 869\n    188 / 869\n    189 / 869\n    190 / 869\n    191"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 68be47b5-81e0-446b-a38f-f91c8e7ef831
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"68be47b5-81e0-446b-a38f-f91c8e7ef831\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4e247c1-22d1-4907-aef1-4732274a2f00\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"64d43718b56745fca00cffb612de77a684090b7cfa12a430c9ed721ebf86d3a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c1187ecc-7dc8-45b6-a0a8-c88aea7b7344\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bcb271b8953ffa7b245f413f281c7091920ef5ba9d32f905079deae9208336cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9350, \"end_char_idx\": 10549, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "147 / 869\n    148 / 869\n    149 / 869\n    150 / 869\n    151 / 869\n    152 / 869\n    153 / 869\n    154 / 869\n    155 / 869\n    156 / 869\n    157 / 869\n    158 / 869\n    159 / 869\n    160 / 869\n    161 / 869\n    162 / 869\n    163 / 869\n    164 / 869\n    165 / 869\n    166 / 869\n    167 / 869\n    168 / 869\n    169 / 869\n    170 / 869\n    171 / 869\n    172 / 869\n    173 / 869\n    174 / 869\n    175 / 869\n    176 / 869\n    177 / 869\n    178 / 869\n    179 / 869\n    180 / 869\n    181 / 869\n    182 / 869\n    183 / 869\n    184 / 869\n    185 / 869\n    186 / 869\n    187 / 869\n    188 / 869\n    189 / 869\n    190 / 869\n    191 / 869\n    192 / 869\n    193 / 869\n    194 / 869\n    195 / 869\n    196 / 869\n    197 / 869\n    198 / 869\n    199 / 869\n    200 / 869\n    201 / 869\n    202 / 869\n    203 / 869\n    204 / 869\n    205 / 869\n    206 / 869\n    207 / 869\n    208 / 869\n    209 / 869\n    210 / 869\n    211 / 869\n    212 / 869\n    213 / 869\n    214 / 869\n    215 / 869\n    216 / 869\n    217 / 869\n    218 / 869\n    219 / 869\n    220 / 869\n    221 / 869\n    222 / 869\n    223 / 869\n    224 / 869\n    225 / 869\n    226 / 869\n    227 / 869\n    228 / 869\n    229 / 869\n    230 / 869\n    231 / 869\n    232 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c1187ecc-7dc8-45b6-a0a8-c88aea7b7344
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c1187ecc-7dc8-45b6-a0a8-c88aea7b7344\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"68be47b5-81e0-446b-a38f-f91c8e7ef831\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4736cfe456685e81e7173c6b8c95019b9290f1965b39f052cd9086a64871d2df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"979fabee-763f-4405-b69e-2f131d24a945\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"52555891ac5c03e6ca120301ba820d69a077cdfb20020b5509f70236fcb88660\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9930, \"end_char_idx\": 11133, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "869\n    189 / 869\n    190 / 869\n    191 / 869\n    192 / 869\n    193 / 869\n    194 / 869\n    195 / 869\n    196 / 869\n    197 / 869\n    198 / 869\n    199 / 869\n    200 / 869\n    201 / 869\n    202 / 869\n    203 / 869\n    204 / 869\n    205 / 869\n    206 / 869\n    207 / 869\n    208 / 869\n    209 / 869\n    210 / 869\n    211 / 869\n    212 / 869\n    213 / 869\n    214 / 869\n    215 / 869\n    216 / 869\n    217 / 869\n    218 / 869\n    219 / 869\n    220 / 869\n    221 / 869\n    222 / 869\n    223 / 869\n    224 / 869\n    225 / 869\n    226 / 869\n    227 / 869\n    228 / 869\n    229 / 869\n    230 / 869\n    231 / 869\n    232 / 869\n    233 / 869\n    234 / 869\n    235 / 869\n    236 / 869\n    237 / 869\n    238 / 869\n    239 / 869\n    240 / 869\n    241 / 869\n    242 / 869\n    243 / 869\n    244 / 869\n    245 / 869\n    246 / 869\n    247 / 869\n    248 / 869\n    249 / 869\n    250 / 869\n    251 / 869\n    252 / 869\n    253 / 869\n    254 / 869\n    255 / 869\n    256 / 869\n    257 / 869\n    258 / 869\n    259 / 869\n    260 / 869\n    261 / 869\n    262 / 869\n    263 / 869\n    264 / 869\n    265 / 869\n    266 / 869\n    267 / 869\n    268 / 869\n    269 / 869\n    270 / 869\n    271 / 869\n    272 / 869\n    273 / 869\n    274 /"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 979fabee-763f-4405-b69e-2f131d24a945
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"979fabee-763f-4405-b69e-2f131d24a945\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c1187ecc-7dc8-45b6-a0a8-c88aea7b7344\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f90a7384e9345b6fa49056e2daa43adeae907d5682068ef11faf9b649ef071d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0af41682-cfb6-4fe4-acd7-8a9efc844ca1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0030700de747c928a235e445e2c41a386807de3cd66c8da5ca24a6a4b73e7aa3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10512, \"end_char_idx\": 11711, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "230 / 869\n    231 / 869\n    232 / 869\n    233 / 869\n    234 / 869\n    235 / 869\n    236 / 869\n    237 / 869\n    238 / 869\n    239 / 869\n    240 / 869\n    241 / 869\n    242 / 869\n    243 / 869\n    244 / 869\n    245 / 869\n    246 / 869\n    247 / 869\n    248 / 869\n    249 / 869\n    250 / 869\n    251 / 869\n    252 / 869\n    253 / 869\n    254 / 869\n    255 / 869\n    256 / 869\n    257 / 869\n    258 / 869\n    259 / 869\n    260 / 869\n    261 / 869\n    262 / 869\n    263 / 869\n    264 / 869\n    265 / 869\n    266 / 869\n    267 / 869\n    268 / 869\n    269 / 869\n    270 / 869\n    271 / 869\n    272 / 869\n    273 / 869\n    274 / 869\n    275 / 869\n    276 / 869\n    277 / 869\n    278 / 869\n    279 / 869\n    280 / 869\n    281 / 869\n    282 / 869\n    283 / 869\n    284 / 869\n    285 / 869\n    286 / 869\n    287 / 869\n    288 / 869\n    289 / 869\n    290 / 869\n    291 / 869\n    292 / 869\n    293 / 869\n    294 / 869\n    295 / 869\n    296 / 869\n    297 / 869\n    298 / 869\n    299 / 869\n    300 / 869\n    301 / 869\n    302 / 869\n    303 / 869\n    304 / 869\n    305 / 869\n    306 / 869\n    307 / 869\n    308 / 869\n    309 / 869\n    310 / 869\n    311 / 869\n    312 / 869\n    313 / 869\n    314 / 869\n    315 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0af41682-cfb6-4fe4-acd7-8a9efc844ca1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0af41682-cfb6-4fe4-acd7-8a9efc844ca1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"979fabee-763f-4405-b69e-2f131d24a945\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c3bec43d464629304b183030e13a80fb68f611c32c4d5dc7cb9cd717b70d10eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e60deac8-d096-4d52-9524-aa810516483f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93e9195bb4d496b16bc85965c34a26f2ce01e9105738f2e7bc9e1a90d339d5a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11100, \"end_char_idx\": 12299, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "272 / 869\n    273 / 869\n    274 / 869\n    275 / 869\n    276 / 869\n    277 / 869\n    278 / 869\n    279 / 869\n    280 / 869\n    281 / 869\n    282 / 869\n    283 / 869\n    284 / 869\n    285 / 869\n    286 / 869\n    287 / 869\n    288 / 869\n    289 / 869\n    290 / 869\n    291 / 869\n    292 / 869\n    293 / 869\n    294 / 869\n    295 / 869\n    296 / 869\n    297 / 869\n    298 / 869\n    299 / 869\n    300 / 869\n    301 / 869\n    302 / 869\n    303 / 869\n    304 / 869\n    305 / 869\n    306 / 869\n    307 / 869\n    308 / 869\n    309 / 869\n    310 / 869\n    311 / 869\n    312 / 869\n    313 / 869\n    314 / 869\n    315 / 869\n    316 / 869\n    317 / 869\n    318 / 869\n    319 / 869\n    320 / 869\n    321 / 869\n    322 / 869\n    323 / 869\n    324 / 869\n    325 / 869\n    326 / 869\n    327 / 869\n    328 / 869\n    329 / 869\n    330 / 869\n    331 / 869\n    332 / 869\n    333 / 869\n    334 / 869\n    335 / 869\n    336 / 869\n    337 / 869\n    338 / 869\n    339 / 869\n    340 / 869\n    341 / 869\n    342 / 869\n    343 / 869\n    344 / 869\n    345 / 869\n    346 / 869\n    347 / 869\n    348 / 869\n    349 / 869\n    350 / 869\n    351 / 869\n    352 / 869\n    353 / 869\n    354 / 869\n    355 / 869\n    356 / 869\n    357 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e60deac8-d096-4d52-9524-aa810516483f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e60deac8-d096-4d52-9524-aa810516483f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0af41682-cfb6-4fe4-acd7-8a9efc844ca1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a2ca91a95141c49bad54b8ea1b80e499096b55a8ec26746d0683d97c175d2246\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b55c7afe-0b9d-41b6-801c-de919ba3cdc0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1fea04060b673970828412b09b51f969ea47a80ec20e7621126ec93111761c9a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11678, \"end_char_idx\": 12881, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "/ 869\n    314 / 869\n    315 / 869\n    316 / 869\n    317 / 869\n    318 / 869\n    319 / 869\n    320 / 869\n    321 / 869\n    322 / 869\n    323 / 869\n    324 / 869\n    325 / 869\n    326 / 869\n    327 / 869\n    328 / 869\n    329 / 869\n    330 / 869\n    331 / 869\n    332 / 869\n    333 / 869\n    334 / 869\n    335 / 869\n    336 / 869\n    337 / 869\n    338 / 869\n    339 / 869\n    340 / 869\n    341 / 869\n    342 / 869\n    343 / 869\n    344 / 869\n    345 / 869\n    346 / 869\n    347 / 869\n    348 / 869\n    349 / 869\n    350 / 869\n    351 / 869\n    352 / 869\n    353 / 869\n    354 / 869\n    355 / 869\n    356 / 869\n    357 / 869\n    358 / 869\n    359 / 869\n    360 / 869\n    361 / 869\n    362 / 869\n    363 / 869\n    364 / 869\n    365 / 869\n    366 / 869\n    367 / 869\n    368 / 869\n    369 / 869\n    370 / 869\n    371 / 869\n    372 / 869\n    373 / 869\n    374 / 869\n    375 / 869\n    376 / 869\n    377 / 869\n    378 / 869\n    379 / 869\n    380 / 869\n    381 / 869\n    382 / 869\n    383 / 869\n    384 / 869\n    385 / 869\n    386 / 869\n    387 / 869\n    388 / 869\n    389 / 869\n    390 / 869\n    391 / 869\n    392 / 869\n    393 / 869\n    394 / 869\n    395 / 869\n    396 / 869\n    397 / 869\n    398 / 869\n    399"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b55c7afe-0b9d-41b6-801c-de919ba3cdc0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b55c7afe-0b9d-41b6-801c-de919ba3cdc0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e60deac8-d096-4d52-9524-aa810516483f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e905f09fa55356301648850da1648a6ece0699c64e6933663f7df8ca32bee9c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f835f47-6546-41d1-b2df-6a871daad1b2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5d9e63171bf1c3f17841ea92032f13fe24fe572bc35df5ffb2a8f706d9d7cdac\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12262, \"end_char_idx\": 13461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "355 / 869\n    356 / 869\n    357 / 869\n    358 / 869\n    359 / 869\n    360 / 869\n    361 / 869\n    362 / 869\n    363 / 869\n    364 / 869\n    365 / 869\n    366 / 869\n    367 / 869\n    368 / 869\n    369 / 869\n    370 / 869\n    371 / 869\n    372 / 869\n    373 / 869\n    374 / 869\n    375 / 869\n    376 / 869\n    377 / 869\n    378 / 869\n    379 / 869\n    380 / 869\n    381 / 869\n    382 / 869\n    383 / 869\n    384 / 869\n    385 / 869\n    386 / 869\n    387 / 869\n    388 / 869\n    389 / 869\n    390 / 869\n    391 / 869\n    392 / 869\n    393 / 869\n    394 / 869\n    395 / 869\n    396 / 869\n    397 / 869\n    398 / 869\n    399 / 869\n    400 / 869\n    401 / 869\n    402 / 869\n    403 / 869\n    404 / 869\n    405 / 869\n    406 / 869\n    407 / 869\n    408 / 869\n    409 / 869\n    410 / 869\n    411 / 869\n    412 / 869\n    413 / 869\n    414 / 869\n    415 / 869\n    416 / 869\n    417 / 869\n    418 / 869\n    419 / 869\n    420 / 869\n    421 / 869\n    422 / 869\n    423 / 869\n    424 / 869\n    425 / 869\n    426 / 869\n    427 / 869\n    428 / 869\n    429 / 869\n    430 / 869\n    431 / 869\n    432 / 869\n    433 / 869\n    434 / 869\n    435 / 869\n    436 / 869\n    437 / 869\n    438 / 869\n    439 / 869\n    440 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3f835f47-6546-41d1-b2df-6a871daad1b2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3f835f47-6546-41d1-b2df-6a871daad1b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b55c7afe-0b9d-41b6-801c-de919ba3cdc0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2dcb093194ffa4e5b90988995094472a58abe80f1fe0b251b06e9f81077665cb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ca06ff4-b345-45c0-bb77-7d89e1be3b07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"000d74d9da6416046430e2811bde2946efc54717907af290c76fbb2c11c8fcf7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12842, \"end_char_idx\": 14045, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "869\n    397 / 869\n    398 / 869\n    399 / 869\n    400 / 869\n    401 / 869\n    402 / 869\n    403 / 869\n    404 / 869\n    405 / 869\n    406 / 869\n    407 / 869\n    408 / 869\n    409 / 869\n    410 / 869\n    411 / 869\n    412 / 869\n    413 / 869\n    414 / 869\n    415 / 869\n    416 / 869\n    417 / 869\n    418 / 869\n    419 / 869\n    420 / 869\n    421 / 869\n    422 / 869\n    423 / 869\n    424 / 869\n    425 / 869\n    426 / 869\n    427 / 869\n    428 / 869\n    429 / 869\n    430 / 869\n    431 / 869\n    432 / 869\n    433 / 869\n    434 / 869\n    435 / 869\n    436 / 869\n    437 / 869\n    438 / 869\n    439 / 869\n    440 / 869\n    441 / 869\n    442 / 869\n    443 / 869\n    444 / 869\n    445 / 869\n    446 / 869\n    447 / 869\n    448 / 869\n    449 / 869\n    450 / 869\n    451 / 869\n    452 / 869\n    453 / 869\n    454 / 869\n    455 / 869\n    456 / 869\n    457 / 869\n    458 / 869\n    459 / 869\n    460 / 869\n    461 / 869\n    462 / 869\n    463 / 869\n    464 / 869\n    465 / 869\n    466 / 869\n    467 / 869\n    468 / 869\n    469 / 869\n    470 / 869\n    471 / 869\n    472 / 869\n    473 / 869\n    474 / 869\n    475 / 869\n    476 / 869\n    477 / 869\n    478 / 869\n    479 / 869\n    480 / 869\n    481 / 869\n    482 /"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2ca06ff4-b345-45c0-bb77-7d89e1be3b07
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2ca06ff4-b345-45c0-bb77-7d89e1be3b07\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f835f47-6546-41d1-b2df-6a871daad1b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e17f6804bfc332c2229d7c8f71b2140e33eab97b22316a83d6d113838d458a8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"89bcdacf-dbbe-43f3-b776-8d871057112d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8ea29681746367447db2c71acc820750c9092636630b7d6a0bdb5d1130e7c76f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13424, \"end_char_idx\": 14623, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "438 / 869\n    439 / 869\n    440 / 869\n    441 / 869\n    442 / 869\n    443 / 869\n    444 / 869\n    445 / 869\n    446 / 869\n    447 / 869\n    448 / 869\n    449 / 869\n    450 / 869\n    451 / 869\n    452 / 869\n    453 / 869\n    454 / 869\n    455 / 869\n    456 / 869\n    457 / 869\n    458 / 869\n    459 / 869\n    460 / 869\n    461 / 869\n    462 / 869\n    463 / 869\n    464 / 869\n    465 / 869\n    466 / 869\n    467 / 869\n    468 / 869\n    469 / 869\n    470 / 869\n    471 / 869\n    472 / 869\n    473 / 869\n    474 / 869\n    475 / 869\n    476 / 869\n    477 / 869\n    478 / 869\n    479 / 869\n    480 / 869\n    481 / 869\n    482 / 869\n    483 / 869\n    484 / 869\n    485 / 869\n    486 / 869\n    487 / 869\n    488 / 869\n    489 / 869\n    490 / 869\n    491 / 869\n    492 / 869\n    493 / 869\n    494 / 869\n    495 / 869\n    496 / 869\n    497 / 869\n    498 / 869\n    499 / 869\n    500 / 869\n    501 / 869\n    502 / 869\n    503 / 869\n    504 / 869\n    505 / 869\n    506 / 869\n    507 / 869\n    508 / 869\n    509 / 869\n    510 / 869\n    511 / 869\n    512 / 869\n    513 / 869\n    514 / 869\n    515 / 869\n    516 / 869\n    517 / 869\n    518 / 869\n    519 / 869\n    520 / 869\n    521 / 869\n    522 / 869\n    523 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 89bcdacf-dbbe-43f3-b776-8d871057112d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"89bcdacf-dbbe-43f3-b776-8d871057112d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ca06ff4-b345-45c0-bb77-7d89e1be3b07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3cebadde37f19e9ba26b5e51b2ff0bacc5e555a5f81148ce6d27b418c6c64cdd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b2ac6c3e-c718-4ec0-8d00-c2f02339f689\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9f4656977b393ed03628a16d6698c146e6a43a6ef9cd3e738e7ed46c06ebafd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14012, \"end_char_idx\": 15211, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "480 / 869\n    481 / 869\n    482 / 869\n    483 / 869\n    484 / 869\n    485 / 869\n    486 / 869\n    487 / 869\n    488 / 869\n    489 / 869\n    490 / 869\n    491 / 869\n    492 / 869\n    493 / 869\n    494 / 869\n    495 / 869\n    496 / 869\n    497 / 869\n    498 / 869\n    499 / 869\n    500 / 869\n    501 / 869\n    502 / 869\n    503 / 869\n    504 / 869\n    505 / 869\n    506 / 869\n    507 / 869\n    508 / 869\n    509 / 869\n    510 / 869\n    511 / 869\n    512 / 869\n    513 / 869\n    514 / 869\n    515 / 869\n    516 / 869\n    517 / 869\n    518 / 869\n    519 / 869\n    520 / 869\n    521 / 869\n    522 / 869\n    523 / 869\n    524 / 869\n    525 / 869\n    526 / 869\n    527 / 869\n    528 / 869\n    529 / 869\n    530 / 869\n    531 / 869\n    532 / 869\n    533 / 869\n    534 / 869\n    535 / 869\n    536 / 869\n    537 / 869\n    538 / 869\n    539 / 869\n    540 / 869\n    541 / 869\n    542 / 869\n    543 / 869\n    544 / 869\n    545 / 869\n    546 / 869\n    547 / 869\n    548 / 869\n    549 / 869\n    550 / 869\n    551 / 869\n    552 / 869\n    553 / 869\n    554 / 869\n    555 / 869\n    556 / 869\n    557 / 869\n    558 / 869\n    559 / 869\n    560 / 869\n    561 / 869\n    562 / 869\n    563 / 869\n    564 / 869\n    565 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b2ac6c3e-c718-4ec0-8d00-c2f02339f689
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b2ac6c3e-c718-4ec0-8d00-c2f02339f689\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"89bcdacf-dbbe-43f3-b776-8d871057112d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"37a56d1fa9fc171ca0770666c223fd4fe7fdd9f866922c064b4d83c1824c84ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c203a501-b8f9-4778-a8e3-f8fa4452a1d7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3aef619707a5e237aae88f166547aa28e65b7f14ae9e3f88b88086cf3640d69a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14590, \"end_char_idx\": 15793, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "/ 869\n    522 / 869\n    523 / 869\n    524 / 869\n    525 / 869\n    526 / 869\n    527 / 869\n    528 / 869\n    529 / 869\n    530 / 869\n    531 / 869\n    532 / 869\n    533 / 869\n    534 / 869\n    535 / 869\n    536 / 869\n    537 / 869\n    538 / 869\n    539 / 869\n    540 / 869\n    541 / 869\n    542 / 869\n    543 / 869\n    544 / 869\n    545 / 869\n    546 / 869\n    547 / 869\n    548 / 869\n    549 / 869\n    550 / 869\n    551 / 869\n    552 / 869\n    553 / 869\n    554 / 869\n    555 / 869\n    556 / 869\n    557 / 869\n    558 / 869\n    559 / 869\n    560 / 869\n    561 / 869\n    562 / 869\n    563 / 869\n    564 / 869\n    565 / 869\n    566 / 869\n    567 / 869\n    568 / 869\n    569 / 869\n    570 / 869\n    571 / 869\n    572 / 869\n    573 / 869\n    574 / 869\n    575 / 869\n    576 / 869\n    577 / 869\n    578 / 869\n    579 / 869\n    580 / 869\n    581 / 869\n    582 / 869\n    583 / 869\n    584 / 869\n    585 / 869\n    586 / 869\n    587 / 869\n    588 / 869\n    589 / 869\n    590 / 869\n    591 / 869\n    592 / 869\n    593 / 869\n    594 / 869\n    595 / 869\n    596 / 869\n    597 / 869\n    598 / 869\n    599 / 869\n    600 / 869\n    601 / 869\n    602 / 869\n    603 / 869\n    604 / 869\n    605 / 869\n    606 / 869\n    607"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c203a501-b8f9-4778-a8e3-f8fa4452a1d7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c203a501-b8f9-4778-a8e3-f8fa4452a1d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b2ac6c3e-c718-4ec0-8d00-c2f02339f689\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d094046043de4ba80681a42cba76fd919381b6f0050bd2277decea160936e9d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a6b248b7-77aa-4adb-9614-14789a3fed48\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4eb7dc84d1966049a89dbd277efbc9735543079e342e3e7697f46ce9beb00042\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15174, \"end_char_idx\": 16373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "563 / 869\n    564 / 869\n    565 / 869\n    566 / 869\n    567 / 869\n    568 / 869\n    569 / 869\n    570 / 869\n    571 / 869\n    572 / 869\n    573 / 869\n    574 / 869\n    575 / 869\n    576 / 869\n    577 / 869\n    578 / 869\n    579 / 869\n    580 / 869\n    581 / 869\n    582 / 869\n    583 / 869\n    584 / 869\n    585 / 869\n    586 / 869\n    587 / 869\n    588 / 869\n    589 / 869\n    590 / 869\n    591 / 869\n    592 / 869\n    593 / 869\n    594 / 869\n    595 / 869\n    596 / 869\n    597 / 869\n    598 / 869\n    599 / 869\n    600 / 869\n    601 / 869\n    602 / 869\n    603 / 869\n    604 / 869\n    605 / 869\n    606 / 869\n    607 / 869\n    608 / 869\n    609 / 869\n    610 / 869\n    611 / 869\n    612 / 869\n    613 / 869\n    614 / 869\n    615 / 869\n    616 / 869\n    617 / 869\n    618 / 869\n    619 / 869\n    620 / 869\n    621 / 869\n    622 / 869\n    623 / 869\n    624 / 869\n    625 / 869\n    626 / 869\n    627 / 869\n    628 / 869\n    629 / 869\n    630 / 869\n    631 / 869\n    632 / 869\n    633 / 869\n    634 / 869\n    635 / 869\n    636 / 869\n    637 / 869\n    638 / 869\n    639 / 869\n    640 / 869\n    641 / 869\n    642 / 869\n    643 / 869\n    644 / 869\n    645 / 869\n    646 / 869\n    647 / 869\n    648 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a6b248b7-77aa-4adb-9614-14789a3fed48
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a6b248b7-77aa-4adb-9614-14789a3fed48\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c203a501-b8f9-4778-a8e3-f8fa4452a1d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5566f89ce8b192fdd0a1c1e68cebc9cc06cdea971ae29b58d977d98fb80ca8f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"335bda9a-862e-4549-aaf9-9e2f6a5ddcf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c72a880bc404f2724a4e83a2118591fbae761d0802b59fea943537f2791cf1a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15754, \"end_char_idx\": 16957, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "869\n    605 / 869\n    606 / 869\n    607 / 869\n    608 / 869\n    609 / 869\n    610 / 869\n    611 / 869\n    612 / 869\n    613 / 869\n    614 / 869\n    615 / 869\n    616 / 869\n    617 / 869\n    618 / 869\n    619 / 869\n    620 / 869\n    621 / 869\n    622 / 869\n    623 / 869\n    624 / 869\n    625 / 869\n    626 / 869\n    627 / 869\n    628 / 869\n    629 / 869\n    630 / 869\n    631 / 869\n    632 / 869\n    633 / 869\n    634 / 869\n    635 / 869\n    636 / 869\n    637 / 869\n    638 / 869\n    639 / 869\n    640 / 869\n    641 / 869\n    642 / 869\n    643 / 869\n    644 / 869\n    645 / 869\n    646 / 869\n    647 / 869\n    648 / 869\n    649 / 869\n    650 / 869\n    651 / 869\n    652 / 869\n    653 / 869\n    654 / 869\n    655 / 869\n    656 / 869\n    657 / 869\n    658 / 869\n    659 / 869\n    660 / 869\n    661 / 869\n    662 / 869\n    663 / 869\n    664 / 869\n    665 / 869\n    666 / 869\n    667 / 869\n    668 / 869\n    669 / 869\n    670 / 869\n    671 / 869\n    672 / 869\n    673 / 869\n    674 / 869\n    675 / 869\n    676 / 869\n    677 / 869\n    678 / 869\n    679 / 869\n    680 / 869\n    681 / 869\n    682 / 869\n    683 / 869\n    684 / 869\n    685 / 869\n    686 / 869\n    687 / 869\n    688 / 869\n    689 / 869\n    690 /"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 335bda9a-862e-4549-aaf9-9e2f6a5ddcf5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"335bda9a-862e-4549-aaf9-9e2f6a5ddcf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6b248b7-77aa-4adb-9614-14789a3fed48\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3d3f4fbe5b0bf627cc6eb117d0fc98c03a330e136174c49b4077543400dcecfe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a206ee18-110d-40d4-b22d-476981e65e08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a0b1434975fec54653bbbdafec9c1de685047f9e4f3b0bfc9f41c32f83c5fbef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16336, \"end_char_idx\": 17535, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "646 / 869\n    647 / 869\n    648 / 869\n    649 / 869\n    650 / 869\n    651 / 869\n    652 / 869\n    653 / 869\n    654 / 869\n    655 / 869\n    656 / 869\n    657 / 869\n    658 / 869\n    659 / 869\n    660 / 869\n    661 / 869\n    662 / 869\n    663 / 869\n    664 / 869\n    665 / 869\n    666 / 869\n    667 / 869\n    668 / 869\n    669 / 869\n    670 / 869\n    671 / 869\n    672 / 869\n    673 / 869\n    674 / 869\n    675 / 869\n    676 / 869\n    677 / 869\n    678 / 869\n    679 / 869\n    680 / 869\n    681 / 869\n    682 / 869\n    683 / 869\n    684 / 869\n    685 / 869\n    686 / 869\n    687 / 869\n    688 / 869\n    689 / 869\n    690 / 869\n    691 / 869\n    692 / 869\n    693 / 869\n    694 / 869\n    695 / 869\n    696 / 869\n    697 / 869\n    698 / 869\n    699 / 869\n    700 / 869\n    701 / 869\n    702 / 869\n    703 / 869\n    704 / 869\n    705 / 869\n    706 / 869\n    707 / 869\n    708 / 869\n    709 / 869\n    710 / 869\n    711 / 869\n    712 / 869\n    713 / 869\n    714 / 869\n    715 / 869\n    716 / 869\n    717 / 869\n    718 / 869\n    719 / 869\n    720 / 869\n    721 / 869\n    722 / 869\n    723 / 869\n    724 / 869\n    725 / 869\n    726 / 869\n    727 / 869\n    728 / 869\n    729 / 869\n    730 / 869\n    731 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a206ee18-110d-40d4-b22d-476981e65e08
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a206ee18-110d-40d4-b22d-476981e65e08\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"335bda9a-862e-4549-aaf9-9e2f6a5ddcf5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e8047929afc51f31a7d810bc096350fcbd600147b1679ab55b103e09dc0b6861\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"420e5d60-7fb4-4caf-9f9a-ab539a5de115\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"36e8e598e303b92e9180b4792018d84110ba0d7e356f5681466eddbadf0028eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16924, \"end_char_idx\": 18123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "688 / 869\n    689 / 869\n    690 / 869\n    691 / 869\n    692 / 869\n    693 / 869\n    694 / 869\n    695 / 869\n    696 / 869\n    697 / 869\n    698 / 869\n    699 / 869\n    700 / 869\n    701 / 869\n    702 / 869\n    703 / 869\n    704 / 869\n    705 / 869\n    706 / 869\n    707 / 869\n    708 / 869\n    709 / 869\n    710 / 869\n    711 / 869\n    712 / 869\n    713 / 869\n    714 / 869\n    715 / 869\n    716 / 869\n    717 / 869\n    718 / 869\n    719 / 869\n    720 / 869\n    721 / 869\n    722 / 869\n    723 / 869\n    724 / 869\n    725 / 869\n    726 / 869\n    727 / 869\n    728 / 869\n    729 / 869\n    730 / 869\n    731 / 869\n    732 / 869\n    733 / 869\n    734 / 869\n    735 / 869\n    736 / 869\n    737 / 869\n    738 / 869\n    739 / 869\n    740 / 869\n    741 / 869\n    742 / 869\n    743 / 869\n    744 / 869\n    745 / 869\n    746 / 869\n    747 / 869\n    748 / 869\n    749 / 869\n    750 / 869\n    751 / 869\n    752 / 869\n    753 / 869\n    754 / 869\n    755 / 869\n    756 / 869\n    757 / 869\n    758 / 869\n    759 / 869\n    760 / 869\n    761 / 869\n    762 / 869\n    763 / 869\n    764 / 869\n    765 / 869\n    766 / 869\n    767 / 869\n    768 / 869\n    769 / 869\n    770 / 869\n    771 / 869\n    772 / 869\n    773 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 420e5d60-7fb4-4caf-9f9a-ab539a5de115
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"420e5d60-7fb4-4caf-9f9a-ab539a5de115\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a206ee18-110d-40d4-b22d-476981e65e08\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"238a47b6e784d80d371c31d72943f189b648ae29f21153d281cd4998e0da1c01\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9dbd0177-51af-40e0-baef-45d6813d120c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"350db7f546bcaa37594aa96547962839bd4713310c25fcac605bd24297ada823\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17502, \"end_char_idx\": 18705, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "/ 869\n    730 / 869\n    731 / 869\n    732 / 869\n    733 / 869\n    734 / 869\n    735 / 869\n    736 / 869\n    737 / 869\n    738 / 869\n    739 / 869\n    740 / 869\n    741 / 869\n    742 / 869\n    743 / 869\n    744 / 869\n    745 / 869\n    746 / 869\n    747 / 869\n    748 / 869\n    749 / 869\n    750 / 869\n    751 / 869\n    752 / 869\n    753 / 869\n    754 / 869\n    755 / 869\n    756 / 869\n    757 / 869\n    758 / 869\n    759 / 869\n    760 / 869\n    761 / 869\n    762 / 869\n    763 / 869\n    764 / 869\n    765 / 869\n    766 / 869\n    767 / 869\n    768 / 869\n    769 / 869\n    770 / 869\n    771 / 869\n    772 / 869\n    773 / 869\n    774 / 869\n    775 / 869\n    776 / 869\n    777 / 869\n    778 / 869\n    779 / 869\n    780 / 869\n    781 / 869\n    782 / 869\n    783 / 869\n    784 / 869\n    785 / 869\n    786 / 869\n    787 / 869\n    788 / 869\n    789 / 869\n    790 / 869\n    791 / 869\n    792 / 869\n    793 / 869\n    794 / 869\n    795 / 869\n    796 / 869\n    797 / 869\n    798 / 869\n    799 / 869\n    800 / 869\n    801 / 869\n    802 / 869\n    803 / 869\n    804 / 869\n    805 / 869\n    806 / 869\n    807 / 869\n    808 / 869\n    809 / 869\n    810 / 869\n    811 / 869\n    812 / 869\n    813 / 869\n    814 / 869\n    815"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9dbd0177-51af-40e0-baef-45d6813d120c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9dbd0177-51af-40e0-baef-45d6813d120c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"420e5d60-7fb4-4caf-9f9a-ab539a5de115\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"69a74db1fe9d69d76d59281bee46aa76657db1e6bdede744e8772f3667424a83\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a597b085-2c6d-4240-b794-4044f3558d6d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dcfe5736b44210ff7dd9bef0d431aed242da6caa4dc6610586e8d0a185b92c07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18086, \"end_char_idx\": 19285, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "771 / 869\n    772 / 869\n    773 / 869\n    774 / 869\n    775 / 869\n    776 / 869\n    777 / 869\n    778 / 869\n    779 / 869\n    780 / 869\n    781 / 869\n    782 / 869\n    783 / 869\n    784 / 869\n    785 / 869\n    786 / 869\n    787 / 869\n    788 / 869\n    789 / 869\n    790 / 869\n    791 / 869\n    792 / 869\n    793 / 869\n    794 / 869\n    795 / 869\n    796 / 869\n    797 / 869\n    798 / 869\n    799 / 869\n    800 / 869\n    801 / 869\n    802 / 869\n    803 / 869\n    804 / 869\n    805 / 869\n    806 / 869\n    807 / 869\n    808 / 869\n    809 / 869\n    810 / 869\n    811 / 869\n    812 / 869\n    813 / 869\n    814 / 869\n    815 / 869\n    816 / 869\n    817 / 869\n    818 / 869\n    819 / 869\n    820 / 869\n    821 / 869\n    822 / 869\n    823 / 869\n    824 / 869\n    825 / 869\n    826 / 869\n    827 / 869\n    828 / 869\n    829 / 869\n    830 / 869\n    831 / 869\n    832 / 869\n    833 / 869\n    834 / 869\n    835 / 869\n    836 / 869\n    837 / 869\n    838 / 869\n    839 / 869\n    840 / 869\n    841 / 869\n    842 / 869\n    843 / 869\n    844 / 869\n    845 / 869\n    846 / 869\n    847 / 869\n    848 / 869\n    849 / 869\n    850 / 869\n    851 / 869\n    852 / 869\n    853 / 869\n    854 / 869\n    855 / 869\n    856 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a597b085-2c6d-4240-b794-4044f3558d6d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb", "title": "Translate a book written in LaTeX from Slovenian into English", "tokens": 8114, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a597b085-2c6d-4240-b794-4044f3558d6d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"98599bee-0fa7-41f3-8a4a-714499130127\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c91bf3d2ea53be7cc51dca3df0981f4f1393c30f032ee61d72223235e01dd67\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9dbd0177-51af-40e0-baef-45d6813d120c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/book_translation/translate_latex_book.ipynb\", \"title\": \"Translate a book written in LaTeX from Slovenian into English\", \"tokens\": 8114, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"10d7be29e4779d0fa59bb32a3d1b2c0666eb30c83edd527b98b59b3124f2ddb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18666, \"end_char_idx\": 19453, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "98599bee-0fa7-41f3-8a4a-714499130127", "doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "ref_doc_id": "98599bee-0fa7-41f3-8a4a-714499130127", "chroma:document": "869\n    813 / 869\n    814 / 869\n    815 / 869\n    816 / 869\n    817 / 869\n    818 / 869\n    819 / 869\n    820 / 869\n    821 / 869\n    822 / 869\n    823 / 869\n    824 / 869\n    825 / 869\n    826 / 869\n    827 / 869\n    828 / 869\n    829 / 869\n    830 / 869\n    831 / 869\n    832 / 869\n    833 / 869\n    834 / 869\n    835 / 869\n    836 / 869\n    837 / 869\n    838 / 869\n    839 / 869\n    840 / 869\n    841 / 869\n    842 / 869\n    843 / 869\n    844 / 869\n    845 / 869\n    846 / 869\n    847 / 869\n    848 / 869\n    849 / 869\n    850 / 869\n    851 / 869\n    852 / 869\n    853 / 869\n    854 / 869\n    855 / 869\n    856 / 869\n    857 / 869\n    858 / 869\n    859 / 869\n    860 / 869\n    861 / 869\n    862 / 869\n    863 / 869\n    864 / 869\n    865 / 869\n    866 / 869\n    867 / 869\n    868 / 869"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cada027e-baf5-45dd-8d54-4c4d4ab73efd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cada027e-baf5-45dd-8d54-4c4d4ab73efd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"02eb5d1d-aa95-4497-b313-a04566f1ce77\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"449b3241257ca1cb72749d8fed1cfd30ec0751c3a75e96bf390c9bc867ecbb9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2466, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>\n\n# 3. Train a fine-tuning model specialized for Q&A\nThis notebook will utilize the dataset of context, question and answer pairs to additionally create adversarial questions and context pairs, where the question was not generated on that context. In those cases the model will be prompted to answer \"No sufficient context for answering the question\". We will also train a discriminator model, which predicts whether the question can be answered based on the context or not.\n\nWe will add hard adversarial examples as well, which will be based either on semantically similar sections, or neighbouring sections, originating from the same article.\n\n\n```python\nimport openai\nimport pandas as pd\ndf = pd.read_csv('olympics-data/olympics_qa.csv')\nolympics_search_fileid = \"file-c3shd8wqF3vSCKaukW4Jr1TT\"\ndf.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>heading</th>\n      <th>content</th>\n      <th>tokens</th>\n      <th>context</th>\n      <th>questions</th>\n      <th>answers</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2020 Summer Olympics</td>\n      <td>Summary</td>\n      <td>The 2020 Summer Olympics (Japanese: 2020\u5e74\u590f\u5b63\u30aa\u30ea\u30f3...</td>\n      <td>713</td>\n      <td>2020 Summer Olympics\\nSummary\\n\\nThe 2020 Summ...</td>\n      <td>1. What is the 2020 Summer Olympics?\\n2. When ...</td>\n      <td>1. The 2020 Summer Olympics is an internationa...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2020 Summer Olympics</td>\n      <td>Host city selection</td>\n      <td>The International Olympic Committee (IOC) vote...</td>\n      <td>126</td>\n      <td>2020 Summer Olympics\\nHost city selection\\n\\nT...</td>\n      <td>1. \\n2. \\n3. \\n4.</td>\n      <td>1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 02eb5d1d-aa95-4497-b313-a04566f1ce77
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"02eb5d1d-aa95-4497-b313-a04566f1ce77\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cada027e-baf5-45dd-8d54-4c4d4ab73efd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"fe1ea95b55a97e61561fa3199aca5e91efa8b2129d5f8eb490231feb12910bf9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3ef14993-3057-4fee-9620-89e823671128\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f12c67abb246a9f846d950c368a933c933fe9360d9809f903f302b6f620653cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1276, \"end_char_idx\": 3670, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>heading</th>\n      <th>content</th>\n      <th>tokens</th>\n      <th>context</th>\n      <th>questions</th>\n      <th>answers</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2020 Summer Olympics</td>\n      <td>Summary</td>\n      <td>The 2020 Summer Olympics (Japanese: 2020\u5e74\u590f\u5b63\u30aa\u30ea\u30f3...</td>\n      <td>713</td>\n      <td>2020 Summer Olympics\\nSummary\\n\\nThe 2020 Summ...</td>\n      <td>1. What is the 2020 Summer Olympics?\\n2. When ...</td>\n      <td>1. The 2020 Summer Olympics is an internationa...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2020 Summer Olympics</td>\n      <td>Host city selection</td>\n      <td>The International Olympic Committee (IOC) vote...</td>\n      <td>126</td>\n      <td>2020 Summer Olympics\\nHost city selection\\n\\nT...</td>\n      <td>1. \\n2. \\n3. \\n4.</td>\n      <td>1. What is the International Olympic Committee...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2020 Summer Olympics</td>\n      <td>Impact of the COVID-19 pandemic</td>\n      <td>In January 2020, concerns were raised about th...</td>\n      <td>369</td>\n      <td>2020 Summer Olympics\\nImpact of the COVID-19 p...</td>\n      <td>1. What was the COVID-19 pandemic?\\n2. How did...</td>\n      <td>1. The COVID-19 pandemic was a pandemic that o...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2020 Summer Olympics</td>\n      <td>Qualifying event cancellation and postponement</td>\n      <td>Concerns about the pandemic began to affect qu...</td>\n      <td>298</td>\n      <td>2020 Summer Olympics\\nQualifying event cancell...</td>\n      <td>1. What was the original location of the Asia ...</td>\n      <td>1. The original location of the Asia &amp; Oceania...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2020 Summer Olympics</td>\n      <td>Effect on doping tests</td>\n      <td>Mandatory doping tests were being severely res...</td>\n      <td>163</td>\n      <td>2020 Summer Olympics\\nEffect on doping tests\\n...</td>\n      <td>1. What was the COVID-19 pandemic?\\n2. What di...</td>\n      <td>1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3ef14993-3057-4fee-9620-89e823671128
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3ef14993-3057-4fee-9620-89e823671128\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"02eb5d1d-aa95-4497-b313-a04566f1ce77\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2949477387ea6d2a2ccf559fbca442dcff98aa6e33a856a1e187b050856607bd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"988fd62a-17ac-447b-a3db-03bfed8c1f85\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"85546658aff985b4d27e4eee28ab43cccb8c549be21360871bda44c87310aeed\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2434, \"end_char_idx\": 5075, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "\\n2. \\n3. \\n4.</td>\n      <td>1. What is the International Olympic Committee...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2020 Summer Olympics</td>\n      <td>Impact of the COVID-19 pandemic</td>\n      <td>In January 2020, concerns were raised about th...</td>\n      <td>369</td>\n      <td>2020 Summer Olympics\\nImpact of the COVID-19 p...</td>\n      <td>1. What was the COVID-19 pandemic?\\n2. How did...</td>\n      <td>1. The COVID-19 pandemic was a pandemic that o...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2020 Summer Olympics</td>\n      <td>Qualifying event cancellation and postponement</td>\n      <td>Concerns about the pandemic began to affect qu...</td>\n      <td>298</td>\n      <td>2020 Summer Olympics\\nQualifying event cancell...</td>\n      <td>1. What was the original location of the Asia ...</td>\n      <td>1. The original location of the Asia &amp; Oceania...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2020 Summer Olympics</td>\n      <td>Effect on doping tests</td>\n      <td>Mandatory doping tests were being severely res...</td>\n      <td>163</td>\n      <td>2020 Summer Olympics\\nEffect on doping tests\\n...</td>\n      <td>1. What was the COVID-19 pandemic?\\n2. What di...</td>\n      <td>1. The COVID-19 pandemic was a pandemic that o...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\nSplit the sections into a training and testing set\n\n\n```python\nfrom sklearn.model_selection import train_test_split\ntrain_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\nlen(train_df), len(test_df)\n```\n\n\n\n\n    (3014, 754)\n\n\n\nwe check that the separator we intend to use isn't present within the contexts\n\n\n```python\ndf.context.str.contains('->').sum()\n```\n\n\n\n\n    0\n\n\n\n## 3.1 Create the fine-tuning datasets for Q&A and discriminator models\nThe fine-tuning dataset is created in the following way. For every corresponding question, answer and context pair we create:\n- Positive example: correct question, answer, context pair\n- Negative examples:\n  - random negative example, where the random context is paired with the question \n  - two hard negative examples\n    - one originating from the same wikipedia article\n    - another, which is most similar to the correct context\n\nThis process is noisy, as sometimes the question might be answerable given a different context, but on average we hope this won't affect the performance too much.\n\nWe apply the same process of dataset creation for both the discriminator, and the Q&A answering model. We apply the process separately for the training and testing set, to ensure that the examples from the training set don't feature within the test set."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 988fd62a-17ac-447b-a3db-03bfed8c1f85
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"988fd62a-17ac-447b-a3db-03bfed8c1f85\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3ef14993-3057-4fee-9620-89e823671128\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfee6d244999dae196f5ee00eaff1e14b42592b913f214a1b0408bf9f2734b5a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"585a6bf8-e695-46ed-98a2-376d4ad7ad96\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"733bf2b599de79aef180c1639506e25efc36cd90590dc885243e4fe29e061b32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3606, \"end_char_idx\": 6839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "What was the COVID-19 pandemic?\\n2. What di...</td>\n      <td>1. The COVID-19 pandemic was a pandemic that o...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\nSplit the sections into a training and testing set\n\n\n```python\nfrom sklearn.model_selection import train_test_split\ntrain_df, test_df = train_test_split(df, test_size=0.2, random_state=42)\nlen(train_df), len(test_df)\n```\n\n\n\n\n    (3014, 754)\n\n\n\nwe check that the separator we intend to use isn't present within the contexts\n\n\n```python\ndf.context.str.contains('->').sum()\n```\n\n\n\n\n    0\n\n\n\n## 3.1 Create the fine-tuning datasets for Q&A and discriminator models\nThe fine-tuning dataset is created in the following way. For every corresponding question, answer and context pair we create:\n- Positive example: correct question, answer, context pair\n- Negative examples:\n  - random negative example, where the random context is paired with the question \n  - two hard negative examples\n    - one originating from the same wikipedia article\n    - another, which is most similar to the correct context\n\nThis process is noisy, as sometimes the question might be answerable given a different context, but on average we hope this won't affect the performance too much.\n\nWe apply the same process of dataset creation for both the discriminator, and the Q&A answering model. We apply the process separately for the training and testing set, to ensure that the examples from the training set don't feature within the test set.\n\n\n```python\nimport random\n\ndef get_random_similar_contexts(question, context, file_id=olympics_search_fileid, search_model='ada', max_rerank=10):\n    \"\"\"\n    Find similar contexts to the given context using the search file\n    \"\"\"\n    try:\n        # TODO: openai.Engine(search_model) is deprecated\n        results = openai.Engine(search_model).search(\n            search_model=search_model, \n            query=question, \n            max_rerank=max_rerank,\n            file=file_id\n        )\n        candidates = []\n        for result in results['data'][:3]:\n            if result['text'] == context:\n                continue\n            candidates.append(result['text'])\n        random_candidate = random.choice(candidates)\n        return random_candidate\n    except Exception as e:\n        print(e)\n        return \"\"\n\ndef create_fine_tuning_dataset(df, discriminator=False, n_negative=1, add_related=False):\n    \"\"\"\n    Create a dataset for fine tuning the OpenAI model; either for a discriminator model, \n    or a model specializing in Q&A, where it says if no relevant context is found.\n\n    Parameters\n    ----------\n    df: pd.DataFrame\n        The dataframe containing the question, answer and context pairs\n    discriminator: bool\n        Whether to create a dataset for the discriminator\n    n_negative: int\n        The number of random negative samples to add (using a random context)\n    add_related: bool\n        Whether to add the related contexts to the correct context. These are hard negative examples\n\n    Returns\n    -------\n    pd.DataFrame\n        The dataframe containing the prompts and completions, ready for fine-tuning\n    \"\"\"\n    rows = []\n    for i, row in df.iterrows():\n        for q, a in zip((\"1.\" + row.questions).split('\\n'), (\"1.\""}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 585a6bf8-e695-46ed-98a2-376d4ad7ad96
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"585a6bf8-e695-46ed-98a2-376d4ad7ad96\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"988fd62a-17ac-447b-a3db-03bfed8c1f85\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"094c80c096530beb2bdcdc379448448f9b619bdaeb472e2c08883f3fb7aa596a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eedb5a93-d4bd-41da-836d-1c14df8d62b2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7cd47454946ab6139afae71ab201585ccc7d5c8834aa578c5fc36a089c5db99f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5078, \"end_char_idx\": 7275, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "```python\nimport random\n\ndef get_random_similar_contexts(question, context, file_id=olympics_search_fileid, search_model='ada', max_rerank=10):\n    \"\"\"\n    Find similar contexts to the given context using the search file\n    \"\"\"\n    try:\n        # TODO: openai.Engine(search_model) is deprecated\n        results = openai.Engine(search_model).search(\n            search_model=search_model, \n            query=question, \n            max_rerank=max_rerank,\n            file=file_id\n        )\n        candidates = []\n        for result in results['data'][:3]:\n            if result['text'] == context:\n                continue\n            candidates.append(result['text'])\n        random_candidate = random.choice(candidates)\n        return random_candidate\n    except Exception as e:\n        print(e)\n        return \"\"\n\ndef create_fine_tuning_dataset(df, discriminator=False, n_negative=1, add_related=False):\n    \"\"\"\n    Create a dataset for fine tuning the OpenAI model; either for a discriminator model, \n    or a model specializing in Q&A, where it says if no relevant context is found.\n\n    Parameters\n    ----------\n    df: pd.DataFrame\n        The dataframe containing the question, answer and context pairs\n    discriminator: bool\n        Whether to create a dataset for the discriminator\n    n_negative: int\n        The number of random negative samples to add (using a random context)\n    add_related: bool\n        Whether to add the related contexts to the correct context. These are hard negative examples\n\n    Returns\n    -------\n    pd.DataFrame\n        The dataframe containing the prompts and completions, ready for fine-tuning\n    \"\"\"\n    rows = []\n    for i, row in df.iterrows():\n        for q, a in zip((\"1.\" + row.questions).split('\\n'), (\"1.\" + row.answers).split('\\n')):\n            if len(q) >10 and len(a) >10:\n                if discriminator:\n                    rows.append({\"prompt\":f\"{row.context}\\nQuestion: {q[2:].strip()}\\n Related:\", \"completion\":f\" yes\"})\n                else:\n                    rows.append({\"prompt\":f\"{row.context}\\nQuestion: {q[2:].strip()}\\nAnswer:\", \"completion\":f\" {a[2:].strip()}\"})\n\n    for i, row in df.iterrows():\n        for q in (\"1.\""}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: eedb5a93-d4bd-41da-836d-1c14df8d62b2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"eedb5a93-d4bd-41da-836d-1c14df8d62b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"585a6bf8-e695-46ed-98a2-376d4ad7ad96\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a11259f49f3a508b1fec420d7a390596c7d84221a3dfc21c0fcf1299e301c4c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"44555f11-8cf8-4c52-af13-d81773941116\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4952c8c96d01674ece45e094eebc13b338fd2ce070d706a4f9c1675cac1e3b7b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6171, \"end_char_idx\": 9452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "Parameters\n    ----------\n    df: pd.DataFrame\n        The dataframe containing the question, answer and context pairs\n    discriminator: bool\n        Whether to create a dataset for the discriminator\n    n_negative: int\n        The number of random negative samples to add (using a random context)\n    add_related: bool\n        Whether to add the related contexts to the correct context. These are hard negative examples\n\n    Returns\n    -------\n    pd.DataFrame\n        The dataframe containing the prompts and completions, ready for fine-tuning\n    \"\"\"\n    rows = []\n    for i, row in df.iterrows():\n        for q, a in zip((\"1.\" + row.questions).split('\\n'), (\"1.\" + row.answers).split('\\n')):\n            if len(q) >10 and len(a) >10:\n                if discriminator:\n                    rows.append({\"prompt\":f\"{row.context}\\nQuestion: {q[2:].strip()}\\n Related:\", \"completion\":f\" yes\"})\n                else:\n                    rows.append({\"prompt\":f\"{row.context}\\nQuestion: {q[2:].strip()}\\nAnswer:\", \"completion\":f\" {a[2:].strip()}\"})\n\n    for i, row in df.iterrows():\n        for q in (\"1.\" + row.questions).split('\\n'):\n            if len(q) >10:\n                for j in range(n_negative + (2 if add_related else 0)):\n                    random_context = \"\"\n                    if j == 0 and add_related:\n                        # add the related contexts based on originating from the same wikipedia page\n                        subset = df[(df.title == row.title) & (df.context != row.context)]\n                        \n                        if len(subset) < 1:\n                            continue\n                        random_context = subset.sample(1).iloc[0].context\n                    if j == 1 and add_related:\n                        # add the related contexts based on the most similar contexts according to the search\n                        random_context = get_random_similar_contexts(q[2:].strip(), row.context, search_model='ada', max_rerank=10)\n                    else:\n                        while True:\n                            # add random context, which isn't the correct context\n                            random_context = df.sample(1).iloc[0].context\n                            if random_context != row.context:\n                                break\n                    if discriminator:\n                        rows.append({\"prompt\":f\"{random_context}\\nQuestion: {q[2:].strip()}\\n Related:\", \"completion\":f\" no\"})\n                    else:\n                        rows.append({\"prompt\":f\"{random_context}\\nQuestion: {q[2:].strip()}\\nAnswer:\", \"completion\":f\" No appropriate context found to answer the question.\"})\n\n    return pd.DataFrame(rows) \n```\n\nWe apply the same process of dataset creation for both the discriminator, and the Q&A answering model. We apply the process separately for the training and testing set, to ensure that the examples from the training set don't feature within the test set.\n\n\n```python\nfor name, is_disc in [('discriminator', True), ('qa', False)]:\n    for train_test, dt in [('train', train_df), ('test', test_df)]:\n        ft = create_fine_tuning_dataset(dt, discriminator=is_disc, n_negative=1, add_related=True)\n        ft.to_json(f'{name}_{train_test}.jsonl', orient='records', lines=True)\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 44555f11-8cf8-4c52-af13-d81773941116
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"44555f11-8cf8-4c52-af13-d81773941116\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eedb5a93-d4bd-41da-836d-1c14df8d62b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8b596840f033942a20f3aa710e916d8e73526c39b133b4b8e56b473a645f86e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d6d94c4-5e0d-400e-a73e-578bb672469e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"480a487e2559ab1c532fa75f9b253037d81cee8e2b0cdc80cebb6e7fc7199876\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8842, \"end_char_idx\": 11388, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "return pd.DataFrame(rows) \n```\n\nWe apply the same process of dataset creation for both the discriminator, and the Q&A answering model. We apply the process separately for the training and testing set, to ensure that the examples from the training set don't feature within the test set.\n\n\n```python\nfor name, is_disc in [('discriminator', True), ('qa', False)]:\n    for train_test, dt in [('train', train_df), ('test', test_df)]:\n        ft = create_fine_tuning_dataset(dt, discriminator=is_disc, n_negative=1, add_related=True)\n        ft.to_json(f'{name}_{train_test}.jsonl', orient='records', lines=True)\n```\n\n\n\n\n    \n\n\n\nWe formatted the data according to the recommendations from the fine-tuning tool, which is available using\n> openai tools fine_tunes.prepare_data -f qa_train.jsonl\n\nWe highly recommend that you use this tool, which suggests improvements in your data formatting for fine-tuning.\n\n\n## 3.2 Submit the datasets for fine-tuning\n\n\n```python\n!openai api fine_tunes.create -t \"olympics-data/discriminator_train.jsonl\" -v \"olympics-data/discriminator_test.jsonl\" --batch_size 16  --compute_classification_metrics --classification_positive_class \" yes\" --model ada\n```\n\n\n\n\n    \n\n\n\n\n```python\n!openai api fine_tunes.create -t \"olympics-data/qa_train.jsonl\" -v \"olympics-data/qa_test.jsonl\" --batch_size 16\n```\n\n\n\n\n    \n\n\n\n## 3.3 Using the fine-tuned models\n\nWe will now use the fine-tuned discriminator and the fine-tuned Q&A model. By requesting logprobs, we can see how certain the discriminator is in a `yes` vs `no` answer.\n\n\n```python\nft_discriminator = \"curie:ft-openai-internal-2021-08-23-23-58-57\"\nft_qa = \"curie:ft-openai-internal-2021-08-23-17-54-10\"\n\ndef apply_ft_discriminator(context, question, discriminator_model):\n    \"\"\"\n    Apply the fine tuned discriminator to a question, to assess whether it can be answered from the context.\n    \"\"\"\n    prompt = f\"{context}\\nQuestion: {question}\\n Related:\"\n    result = openai.chat.completions.create(model=discriminator_model, prompt=prompt, max_tokens=1, temperature=0, top_p=1, n=1, logprobs=2)\n    return result['choices'][0]['logprobs']['top_logprobs']\n\napply_ft_discriminator('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.', \n                        'What was the first human-made object in space?', ft_discriminator)\n```\n\n\n\n\n    [<OpenAIObject at 0x7fe812e602b0> JSON: {\n       \" no\": -10.819577,\n       \" yes\": -2.045765e-05\n     }]\n\n\n\nWe can see that the model can generalize well to different contexts and questions."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1d6d94c4-5e0d-400e-a73e-578bb672469e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1d6d94c4-5e0d-400e-a73e-578bb672469e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44555f11-8cf8-4c52-af13-d81773941116\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cc9a108cc7e47383d793290276bfeedc246598fdc3b746865ec9bf7da9fc99eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7cfb0270-59cd-4eb8-902c-c59c4b63cf45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"27ff64ed005db3e16c69aebcedb676879b47791f883923d5c60482304732d79e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10037, \"end_char_idx\": 12758, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "```python\n!openai api fine_tunes.create -t \"olympics-data/qa_train.jsonl\" -v \"olympics-data/qa_test.jsonl\" --batch_size 16\n```\n\n\n\n\n    \n\n\n\n## 3.3 Using the fine-tuned models\n\nWe will now use the fine-tuned discriminator and the fine-tuned Q&A model. By requesting logprobs, we can see how certain the discriminator is in a `yes` vs `no` answer.\n\n\n```python\nft_discriminator = \"curie:ft-openai-internal-2021-08-23-23-58-57\"\nft_qa = \"curie:ft-openai-internal-2021-08-23-17-54-10\"\n\ndef apply_ft_discriminator(context, question, discriminator_model):\n    \"\"\"\n    Apply the fine tuned discriminator to a question, to assess whether it can be answered from the context.\n    \"\"\"\n    prompt = f\"{context}\\nQuestion: {question}\\n Related:\"\n    result = openai.chat.completions.create(model=discriminator_model, prompt=prompt, max_tokens=1, temperature=0, top_p=1, n=1, logprobs=2)\n    return result['choices'][0]['logprobs']['top_logprobs']\n\napply_ft_discriminator('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.', \n                        'What was the first human-made object in space?', ft_discriminator)\n```\n\n\n\n\n    [<OpenAIObject at 0x7fe812e602b0> JSON: {\n       \" no\": -10.819577,\n       \" yes\": -2.045765e-05\n     }]\n\n\n\nWe can see that the model can generalize well to different contexts and questions. \n\n\n```python\ndef apply_ft_qa_answer(context, question, answering_model):\n    \"\"\"\n    Apply the fine tuned discriminator to a question\n    \"\"\"\n    prompt = f\"{context}\\nQuestion: {question}\\nAnswer:\"\n    result = openai.chat.completions.create(model=answering_model, prompt=prompt, max_tokens=30, temperature=0, top_p=1, n=1, stop=['.','\\n'])\n    return result['choices'][0]['text']\n\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.', \n                    'What was the first human-made object in space?', ft_qa)\n\n```\n\n\n\n\n    ' The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957'\n\n\n\nWe can see that the model can answer the question, when the context is appropriate.\n\n\n```python\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.',\n                    'What is impressive about the Soviet Union?', ft_qa)\n```\n\n\n\n\n    ' The Soviet Union was the first country to successfully launch a satellite into space'\n\n\n\n\n```python\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.',\n                    'How many cars were produced in the Soviet Union in 1970?', ft_qa)\n```\n\n\n\n\n    ' No appropriate context found to answer the question'"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7cfb0270-59cd-4eb8-902c-c59c4b63cf45
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7cfb0270-59cd-4eb8-902c-c59c4b63cf45\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d6d94c4-5e0d-400e-a73e-578bb672469e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a1c4015503a2d8ebe3f69ed58565773f059e70828ae7ead42a24ace26ddf1034\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31eff65b-e02f-4ab1-bc76-9fff4401647c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd0e2870aa9a41a060aa057792de758d51ceea743c78493f1bdf148ccfa7b3ba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11198, \"end_char_idx\": 14490, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "[<OpenAIObject at 0x7fe812e602b0> JSON: {\n       \" no\": -10.819577,\n       \" yes\": -2.045765e-05\n     }]\n\n\n\nWe can see that the model can generalize well to different contexts and questions. \n\n\n```python\ndef apply_ft_qa_answer(context, question, answering_model):\n    \"\"\"\n    Apply the fine tuned discriminator to a question\n    \"\"\"\n    prompt = f\"{context}\\nQuestion: {question}\\nAnswer:\"\n    result = openai.chat.completions.create(model=answering_model, prompt=prompt, max_tokens=30, temperature=0, top_p=1, n=1, stop=['.','\\n'])\n    return result['choices'][0]['text']\n\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.', \n                    'What was the first human-made object in space?', ft_qa)\n\n```\n\n\n\n\n    ' The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957'\n\n\n\nWe can see that the model can answer the question, when the context is appropriate.\n\n\n```python\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.',\n                    'What is impressive about the Soviet Union?', ft_qa)\n```\n\n\n\n\n    ' The Soviet Union was the first country to successfully launch a satellite into space'\n\n\n\n\n```python\napply_ft_qa_answer('The first human-made object in space was the Soviet Union satellite Sputnik 1 on 4 October 1957.',\n                    'How many cars were produced in the Soviet Union in 1970?', ft_qa)\n```\n\n\n\n\n    ' No appropriate context found to answer the question'\n\n\n\nWe can see that the model knows when to answer the question, and when to say that insufficient context is present to answer the question.\n\nWe can also combine a discriminator and a base model, or a fine-tuned Q&A model. Discriminator can essentially serve as a decision whether the question can be answered given the context or not.\n\n\n```python\ndef answer_question_conditionally(answering_model, discriminator_model, context, question, discriminator_logprob_yes_modifier=0):\n    logprobs = apply_ft_discriminator(context, question, discriminator_model)\n    yes_logprob = logprobs[' yes'] if ' yes' in logprobs else -100\n    no_logprob = logprobs[' no'] if ' no' in logprobs else -100\n    if yes_logprob + discriminator_logprob_yes_modifier < no_logprob:\n        return \" No appropriate context found to answer the question based on the discriminator.\"\n    return apply_ft_qa_answer(context, question, answering_model)\nanswer_question_conditionally(ft_qa, ft_discriminator, \n                                \"Crowdless games are a rare although not unheard-of occurrence in sports. \\\n                                 When they do occur, it is usually the result of events beyond the control \\\n                                 of the teams or fans, such as weather-related concerns, public health concerns, \\\n                                 or wider civil disturbances unrelated to the game. For instance, \\\n                                 the COVID-19 pandemic caused many sports leagues around the world \\\n                                 to be played behind closed doors.\",\n                                \"Could weather cause a sport event to have no crowd?\")\n```\n\n\n\n\n    ' Weather could cause a sport event to have no crowd'"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 31eff65b-e02f-4ab1-bc76-9fff4401647c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb", "title": "3. Train a fine-tuning model specialized for Q&A", "tokens": 3706, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"31eff65b-e02f-4ab1-bc76-9fff4401647c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb37adf2-6f64-4e26-a5fb-cd52c9612a4a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99783ea25e0a6d8e29e03b1df4bf5a4b6e2711938dbea46603a4e74242b391ee\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7cfb0270-59cd-4eb8-902c-c59c4b63cf45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-3-train-qa.ipynb\", \"title\": \"3. Train a fine-tuning model specialized for Q&A\", \"tokens\": 3706, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"87b9a9314bf4faf897e2d8811933664177c0a505ecad5548050705f89cfac717\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12704, \"end_char_idx\": 15566, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "ref_doc_id": "fb37adf2-6f64-4e26-a5fb-cd52c9612a4a", "chroma:document": "' No appropriate context found to answer the question'\n\n\n\nWe can see that the model knows when to answer the question, and when to say that insufficient context is present to answer the question.\n\nWe can also combine a discriminator and a base model, or a fine-tuned Q&A model. Discriminator can essentially serve as a decision whether the question can be answered given the context or not.\n\n\n```python\ndef answer_question_conditionally(answering_model, discriminator_model, context, question, discriminator_logprob_yes_modifier=0):\n    logprobs = apply_ft_discriminator(context, question, discriminator_model)\n    yes_logprob = logprobs[' yes'] if ' yes' in logprobs else -100\n    no_logprob = logprobs[' no'] if ' no' in logprobs else -100\n    if yes_logprob + discriminator_logprob_yes_modifier < no_logprob:\n        return \" No appropriate context found to answer the question based on the discriminator.\"\n    return apply_ft_qa_answer(context, question, answering_model)\nanswer_question_conditionally(ft_qa, ft_discriminator, \n                                \"Crowdless games are a rare although not unheard-of occurrence in sports. \\\n                                 When they do occur, it is usually the result of events beyond the control \\\n                                 of the teams or fans, such as weather-related concerns, public health concerns, \\\n                                 or wider civil disturbances unrelated to the game. For instance, \\\n                                 the COVID-19 pandemic caused many sports leagues around the world \\\n                                 to be played behind closed doors.\",\n                                \"Could weather cause a sport event to have no crowd?\")\n```\n\n\n\n\n    ' Weather could cause a sport event to have no crowd'\n\n\n\nThe above function illustrates how to potentially combine a discriminator and a fine-tuned Q&A model. This gives a more fine-grained control over how certain we want the model to be before it answers the question.\n\nWe'll now take a look on how answers endpoint works - combining search to retrieve the relevant context from a knowledge base, and then using the fine-tuned Q&A model to answer the question.\n\n## 3.4 Answering the question based on a knowledge base\nFinally we can use a logic similar to the [/answers](https://beta.openai.com/docs/api-reference/answers) endpoint, where we first search for the relevant context, and then ask a Q&A model to answer the question given that context. If you'd like to see the implementation details, check out the [`answers_with_ft.py`](answers_with_ft.py) file.\n\n\n```python\nfrom answers_with_ft import answer_question\nanswer_question(olympics_search_fileid, ft_qa, \"Which country won the Women's football tournament at the 2020 Olympic games?\")\n```\n\n\n\n\n    \" Canada won the Women's football tournament at the 2020 Olympic games\""}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 57a1997f-920f-43da-8684-dcb5d216628b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb", "title": "1. Collect Wikipedia data about Olympic Games 2020", "tokens": 2914, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"57a1997f-920f-43da-8684-dcb5d216628b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34d0abca7de45c705e6718c8ac1bf17010d59c5cb4d08c85c144f09f4160314e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2f1e32d9-bb3f-4dc5-9d81-fb345274caa2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"abf35b6ef3cb253f1c316c5be58c6c77bc7ed80f0ffa7eec342ad985636ebf89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2537, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "ref_doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "chroma:document": "<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>\n\n# 1. Collect Wikipedia data about Olympic Games 2020\n\nThe idea of this project is to create a question answering model, based on a few paragraphs of provided text. Base GPT-3 models do a good job at answering questions when the answer is contained within the paragraph, however if the answer isn't contained, the base models tend to try their best to answer anyway, often leading to confabulated answers. \n\nTo create a model which answers questions only if there is sufficient context for doing so, we first create a dataset of questions and answers based on paragraphs of text. In order to train the model to answer only when the answer is present, we also add adversarial examples, where the question doesn't match the context. In those cases, we ask the model to output \"No sufficient context for answering the question\". \n\nWe will perform this task in three notebooks:\n1. The first (this) notebook focuses on collecting recent data, which GPT-3 didn't see during it's pre-training. We picked the topic of Olympic Games 2020 (which actually took place in the summer of 2021), and downloaded 713 unique pages. We organized the dataset by individual sections, which will serve as context for asking and answering the questions.\n2. The [second notebook](olympics-2-create-qa.ipynb) will utilize Davinci-instruct to ask a few questions based on a Wikipedia section, as well as answer those questions, based on that section.\n3. The [third notebook](olympics-3-train-qa.ipynb) will utilize the dataset of context, question and answer pairs to additionally create adversarial questions and context pairs, where the question was not generated on that context. In those cases the model will be prompted to answer \"No sufficient context for answering the question\". We will also train a discriminator model, which predicts whether the question can be answered based on the context or not.\n\n## 1.1 Data extraction using the wikipedia API\nExtracting the data will take about half an hour, and processing will likely take about as much.\n\n\n```python\nimport pandas as pd\nimport wikipedia"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2f1e32d9-bb3f-4dc5-9d81-fb345274caa2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb", "title": "1. Collect Wikipedia data about Olympic Games 2020", "tokens": 2914, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2f1e32d9-bb3f-4dc5-9d81-fb345274caa2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34d0abca7de45c705e6718c8ac1bf17010d59c5cb4d08c85c144f09f4160314e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"57a1997f-920f-43da-8684-dcb5d216628b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f01f28b3b8cdc1de9f99908599a3bcebbcd3251d8eb78c382a65a92b10112164\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22c6373d-c67b-4be7-8c06-aef2810e8ec2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a3a087c022e785b48b6f357cb462adc284d1c5d1116f1be227075e044f7641a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2491, \"end_char_idx\": 5383, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "ref_doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "chroma:document": "```python\nimport pandas as pd\nimport wikipedia\n\n\ndef filter_olympic_2020_titles(titles):\n    \"\"\"\n    Get the titles which are related to Olympic games hosted in 2020, given a list of titles\n    \"\"\"\n    titles = [title for title in titles if '2020' in title and 'olympi' in title.lower()]\n    \n    return titles\n\ndef get_wiki_page(title):\n    \"\"\"\n    Get the wikipedia page given a title\n    \"\"\"\n    try:\n        return wikipedia.page(title)\n    except wikipedia.exceptions.DisambiguationError as e:\n        return wikipedia.page(e.options[0])\n    except wikipedia.exceptions.PageError as e:\n        return None\n\ndef recursively_find_all_pages(titles, titles_so_far=set()):\n    \"\"\"\n    Recursively find all the pages that are linked to the Wikipedia titles in the list\n    \"\"\"\n    all_pages = []\n    \n    titles = list(set(titles) - titles_so_far)\n    titles = filter_olympic_2020_titles(titles)\n    titles_so_far.update(titles)\n    for title in titles:\n        page = get_wiki_page(title)\n        if page is None:\n            continue\n        all_pages.append(page)\n\n        new_pages = recursively_find_all_pages(page.links, titles_so_far)\n        for pg in new_pages:\n            if pg.title not in [p.title for p in all_pages]:\n                all_pages.append(pg)\n        titles_so_far.update(page.links)\n    return all_pages\n\n\npages = recursively_find_all_pages([\"2020 Summer Olympics\"])\nlen(pages)\n```\n\n\n\n\n    909\n\n\n\n## 1.2 Filtering the Wikipedia pages and splitting them into sections by headings\nWe remove sections unlikely to contain textual information, and ensure that each section is not longer than the token limit\n\n\n```python\n\nimport re\nfrom typing import Set\nfrom transformers import GPT2TokenizerFast\n\nimport numpy as np\nfrom nltk.tokenize import sent_tokenize\n\ntokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")\n\ndef count_tokens(text: str) -> int:\n    \"\"\"count the number of tokens in a string\"\"\"\n    return len(tokenizer.encode(text))\n\ndef reduce_long(\n    long_text: str, long_text_tokens: bool = False, max_len: int = 590\n) -> str:\n    \"\"\"\n    Reduce a long text to a maximum of `max_len` tokens by potentially cutting at a sentence end\n    \"\"\"\n    if not long_text_tokens:\n        long_text_tokens = count_tokens(long_text)\n    if long_text_tokens > max_len:\n        sentences = sent_tokenize(long_text.replace(\"\\n\", \" \"))\n        ntokens = 0\n        for i, sentence in enumerate(sentences):\n            ntokens += 1 + count_tokens(sentence)\n            if ntokens > max_len:\n                return \". \".join(sentences[:i]) + \".\"\n\n    return long_text\n\ndiscard_categories = ['See also', 'References', 'External links', 'Further reading', \"Footnotes\",\n    \"Bibliography\", \"Sources\", \"Citations\", \"Literature\", \"Footnotes\", \"Notes and references\",\n    \"Photo gallery\", \"Works cited\", \"Photos\", \"Gallery\", \"Notes\", \"References and sources\",\n    \"References and notes\",]"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 22c6373d-c67b-4be7-8c06-aef2810e8ec2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb", "title": "1. Collect Wikipedia data about Olympic Games 2020", "tokens": 2914, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"22c6373d-c67b-4be7-8c06-aef2810e8ec2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34d0abca7de45c705e6718c8ac1bf17010d59c5cb4d08c85c144f09f4160314e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f1e32d9-bb3f-4dc5-9d81-fb345274caa2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0d88a453d5261501010b326e5215c8c19ca02436ad1ff3d1e0f9143bda528308\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cc2ac5a3-de39-463a-bec2-64c76873a577\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56fa6a030c3d7a503e8064cea6a1b7c497fb0554dca98016efbe46ad3037f503\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3823, \"end_char_idx\": 7821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "ref_doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "chroma:document": "pages = recursively_find_all_pages([\"2020 Summer Olympics\"])\nlen(pages)\n```\n\n\n\n\n    909\n\n\n\n## 1.2 Filtering the Wikipedia pages and splitting them into sections by headings\nWe remove sections unlikely to contain textual information, and ensure that each section is not longer than the token limit\n\n\n```python\n\nimport re\nfrom typing import Set\nfrom transformers import GPT2TokenizerFast\n\nimport numpy as np\nfrom nltk.tokenize import sent_tokenize\n\ntokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")\n\ndef count_tokens(text: str) -> int:\n    \"\"\"count the number of tokens in a string\"\"\"\n    return len(tokenizer.encode(text))\n\ndef reduce_long(\n    long_text: str, long_text_tokens: bool = False, max_len: int = 590\n) -> str:\n    \"\"\"\n    Reduce a long text to a maximum of `max_len` tokens by potentially cutting at a sentence end\n    \"\"\"\n    if not long_text_tokens:\n        long_text_tokens = count_tokens(long_text)\n    if long_text_tokens > max_len:\n        sentences = sent_tokenize(long_text.replace(\"\\n\", \" \"))\n        ntokens = 0\n        for i, sentence in enumerate(sentences):\n            ntokens += 1 + count_tokens(sentence)\n            if ntokens > max_len:\n                return \". \".join(sentences[:i]) + \".\"\n\n    return long_text\n\ndiscard_categories = ['See also', 'References', 'External links', 'Further reading', \"Footnotes\",\n    \"Bibliography\", \"Sources\", \"Citations\", \"Literature\", \"Footnotes\", \"Notes and references\",\n    \"Photo gallery\", \"Works cited\", \"Photos\", \"Gallery\", \"Notes\", \"References and sources\",\n    \"References and notes\",]\n\n\ndef extract_sections(\n    wiki_text: str,\n    title: str,\n    max_len: int = 1500,\n    discard_categories: Set[str] = discard_categories,\n) -> str:\n    \"\"\"\n    Extract the sections of a Wikipedia page, discarding the references and other low information sections\n    \"\"\"\n    if len(wiki_text) == 0:\n        return []\n\n    # find all headings and the corresponding contents\n    headings = re.findall(\"==+ .* ==+\", wiki_text)\n    for heading in headings:\n        wiki_text = wiki_text.replace(heading, \"==+ !! ==+\")\n    contents = wiki_text.split(\"==+ !! ==+\")\n    contents = [c.strip() for c in contents]\n    assert len(headings) == len(contents) - 1\n\n    cont = contents.pop(0).strip()\n    outputs = [(title, \"Summary\", cont, count_tokens(cont)+4)]\n    \n    # discard the discard categories, accounting for a tree structure\n    max_level = 100\n    keep_group_level = max_level\n    remove_group_level = max_level\n    nheadings, ncontents = [], []\n    for heading, content in zip(headings, contents):\n        plain_heading = \" \".join(heading.split(\" \")[1:-1])\n        num_equals = len(heading.split(\" \")[0])\n        if num_equals <= keep_group_level:\n            keep_group_level = max_level\n\n        if num_equals > remove_group_level:\n            if (\n                num_equals <= keep_group_level\n            ):\n                continue\n        keep_group_level = max_level\n        if plain_heading in discard_categories:\n            remove_group_level = num_equals\n            keep_group_level = max_level\n            continue\n        nheadings.append(heading.replace(\"=\", \"\").strip())\n        ncontents.append(content)\n        remove_group_level = max_level\n\n    # count the tokens of each section\n    ncontent_ntokens = [\n        count_tokens(c)\n        + 3\n        + count_tokens(\" \".join(h.split(\" \")[1:-1]))\n        - (1 if len(c) == 0 else 0)\n        for h, c in zip(nheadings, ncontents)\n    ]\n\n    # Create a tuple of (title, section_name, content, number of tokens)\n    outputs += [(title, h, c, t) if t<max_len \n                else (title, h, reduce_long(c, max_len), count_tokens(reduce_long(c,max_len))) \n                    for h, c, t in zip(nheadings, ncontents, ncontent_ntokens)]\n    \n    return outputs\n\n# Example page being processed into sections\nbermuda_page = get_wiki_page('Bermuda at the 2020 Summer Olympics')\nber = extract_sections(bermuda_page.content, bermuda_page.title)\n\n# Example section\nber[-1]\n\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cc2ac5a3-de39-463a-bec2-64c76873a577
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb", "title": "1. Collect Wikipedia data about Olympic Games 2020", "tokens": 2914, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cc2ac5a3-de39-463a-bec2-64c76873a577\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34d0abca7de45c705e6718c8ac1bf17010d59c5cb4d08c85c144f09f4160314e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"22c6373d-c67b-4be7-8c06-aef2810e8ec2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e6f1c8904bb26901182de4ca9728bdebf012142ec0948f1a11d12bdeaaec1805\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"55bac25a-ea19-4622-95d3-40fcaf755711\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0074b278e894572f27120ec3f93bd591ddd371d9a6339b288dc192a86b320b81\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7830, \"end_char_idx\": 10514, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "ref_doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "chroma:document": "('Bermuda at the 2020 Summer Olympics',\n     'Equestrian',\n     \"Bermuda entered one dressage rider into the Olympic competition by finishing in the top four, outside the group selection, of the individual FEI Olympic Rankings for Groups D and E (North, Central, and South America), marking the country's recurrence to the sport after an eight-year absence. The quota was later withdrawn, following an injury of Annabelle Collins' main horse Joyero and a failure to obtain minimum eligibility requirements (MER) aboard a new horse Chuppy Checker.\",\n     104)\n\n\n\n### 1.2.1 We create a dataset and filter out any sections with fewer than 40 tokens, as those are unlikely to contain enough context to ask a good question.\n\n\n```python\nres = []\nfor page in pages:\n    res += extract_sections(page.content, page.title)\ndf = pd.DataFrame(res, columns=[\"title\", \"heading\", \"content\", \"tokens\"])\ndf = df[df.tokens>40]\ndf = df.drop_duplicates(['title','heading'])\ndf = df.reset_index().drop('index',axis=1) # reset index\ndf.head()\n```\n\n    Token indices sequence length is longer than the specified maximum sequence length for this model (1060 > 1024). Running this sequence through the model will result in indexing errors\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>heading</th>\n      <th>content</th>\n      <th>tokens</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2020 Summer Olympics</td>\n      <td>Summary</td>\n      <td>The 2020 Summer Olympics (Japanese: 2020\u5e74\u590f\u5b63\u30aa\u30ea\u30f3...</td>\n      <td>713</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2020 Summer Olympics</td>\n      <td>Host city selection</td>\n      <td>The International Olympic Committee (IOC) vote...</td>\n      <td>126</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2020 Summer Olympics</td>\n      <td>Impact of the COVID-19 pandemic</td>\n      <td>In January 2020, concerns were raised about th...</td>\n      <td>369</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2020 Summer Olympics</td>\n      <td>Qualifying event cancellation and postponement</td>\n      <td>Concerns about the pandemic began to affect qu...</td>\n      <td>298</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2020 Summer Olympics</td>\n      <td>Effect on doping tests</td>\n      <td>Mandatory doping tests were being severely res...</td>\n      <td>163</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 55bac25a-ea19-4622-95d3-40fcaf755711
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb", "title": "1. Collect Wikipedia data about Olympic Games 2020", "tokens": 2914, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"55bac25a-ea19-4622-95d3-40fcaf755711\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34d0abca7de45c705e6718c8ac1bf17010d59c5cb4d08c85c144f09f4160314e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc2ac5a3-de39-463a-bec2-64c76873a577\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-1-collect-data.ipynb\", \"title\": \"1. Collect Wikipedia data about Olympic Games 2020\", \"tokens\": 2914, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"52ba93a8fdc27cc152154732f89ee9148466340e95622859aeb1c7b23bc9b155\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10518, \"end_char_idx\": 12062, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "ref_doc_id": "70a5ecc6-28b9-42f5-8fc4-c2bd4f66c8cd", "chroma:document": "### Save the section dataset\nWe will save the section dataset, for the [next notebook](olympics-2-create-qa.ipynb)\n\n\n```python\ndf.to_csv('olympics-data/olympics_sections.csv', index=False)\n```\n\n## 1.3 (Optional) Exploring the data \n\n\n```python\ndf.title.value_counts().head()\n```\n\n\n\n\n    Concerns and controversies at the 2020 Summer Olympics    51\n    United States at the 2020 Summer Olympics                 46\n    Great Britain at the 2020 Summer Olympics                 42\n    Canada at the 2020 Summer Olympics                        39\n    Olympic Games                                             39\n    Name: title, dtype: int64\n\n\n\nThere appear to be winter and summer Olympics 2020. We chose to leave a little ambiguity and noise in the dataset, even though we were interested in only Summer Olympics 2020.\n\n\n```python\ndf.title.str.contains('Summer').value_counts()\n```\n\n\n\n\n    True     3567\n    False     305\n    Name: title, dtype: int64\n\n\n\n\n```python\ndf.title.str.contains('Winter').value_counts()\n```\n\n\n\n\n    False    3774\n    True       98\n    Name: title, dtype: int64\n\n\n\n\n```python\nimport pandas as pd\nfrom matplotlib import pyplot as plt\n\ndf = pd.read_csv('olympics-data/olympics_sections.csv')\ndf[['tokens']].hist()\n# add axis descriptions and title\nplt.xlabel('Number of tokens')\nplt.ylabel('Number of Wikipedia sections')\nplt.title('Distribution of number of tokens in Wikipedia sections')\nplt.show()\n```\n\n\n    \n![png](output_15_0.png)\n    \n\n\nWe can see that the majority of section are fairly short (less than 500 tokens)."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0fa6ab01-8af9-4eb4-a0a0-2374dfcbf153
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0fa6ab01-8af9-4eb4-a0a0-2374dfcbf153\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7a7495e3-4d32-4e63-8e61-6f0435064238\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f898e720fef8a88b50fc010a9994ac9429bfb03f2224422bc1dda50fe6814fe5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1397, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>\n\n# 2. Creating a synthetic Q&A dataset\nWe use [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), a model specialized in following instructions, to create questions based on the given context. Then we also use [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta) to answer those questions, given the same context. \n\nThis is expensive, and will also take a long time, as we call the davinci engine for each section. You can simply download the final dataset instead.\n\nWe're using the dataset created using the [previous notebook](olympics-1-collect-data.ipynb)\n\n## 2.1 Read in the data, and create a context\nCreate a context by concatenating the title, the heading and the content of that section\n\n\n```python\nimport pandas as pd\ndf = pd.read_csv('olympics-data/olympics_sections.csv')\ndf['context'] = df.title + \"\\n\" + df.heading + \"\\n\\n\" + df.content\ndf.head()\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7a7495e3-4d32-4e63-8e61-6f0435064238
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7a7495e3-4d32-4e63-8e61-6f0435064238\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0fa6ab01-8af9-4eb4-a0a0-2374dfcbf153\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bd24a168a6c6f0d4fda1caeae945f850959a36fac1ceeb255c7f47046032211e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de77ba37-44bb-40b3-8360-d637a4dc3638\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd95ab2fbdc31bed7ed03497313e3fea2bd8f3253b0bb22457cffb08619c6ff3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3215, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "<span style=\"color:orange; font-weight:bold\">Note: To answer questions based on text documents, we recommend the procedure in <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering using Embeddings</a>. Some of the code below may rely on <a href=\"https://github.com/openai/openai-cookbook/tree/main/transition_guides_for_deprecated_API_endpoints\">deprecated API endpoints</a>.</span>\n\n# 2. Creating a synthetic Q&A dataset\nWe use [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), a model specialized in following instructions, to create questions based on the given context. Then we also use [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta) to answer those questions, given the same context. \n\nThis is expensive, and will also take a long time, as we call the davinci engine for each section. You can simply download the final dataset instead.\n\nWe're using the dataset created using the [previous notebook](olympics-1-collect-data.ipynb)\n\n## 2.1 Read in the data, and create a context\nCreate a context by concatenating the title, the heading and the content of that section\n\n\n```python\nimport pandas as pd\ndf = pd.read_csv('olympics-data/olympics_sections.csv')\ndf['context'] = df.title + \"\\n\" + df.heading + \"\\n\\n\" + df.content\ndf.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>heading</th>\n      <th>content</th>\n      <th>tokens</th>\n      <th>context</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2020 Summer Olympics</td>\n      <td>Summary</td>\n      <td>The 2020 Summer Olympics (Japanese: 2020\u5e74\u590f\u5b63\u30aa\u30ea\u30f3...</td>\n      <td>713</td>\n      <td>2020 Summer Olympics\\nSummary\\n\\nThe 2020 Summ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2020 Summer Olympics</td>\n      <td>Host city selection</td>\n      <td>The International Olympic Committee (IOC) vote...</td>\n      <td>126</td>\n      <td>2020 Summer Olympics\\nHost city selection\\n\\nT...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2020 Summer Olympics</td>\n      <td>Impact of the COVID-19 pandemic</td>\n      <td>In January 2020, concerns were raised about th...</td>\n      <td>369</td>\n      <td>2020 Summer Olympics\\nImpact of the COVID-19 p...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2020 Summer Olympics</td>\n      <td>Qualifying event cancellation and postponement</td>\n      <td>Concerns about the pandemic began to affect qu...</td>\n      <td>298</td>\n      <td>2020 Summer Olympics\\nQualifying event cancell...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2020 Summer Olympics</td>\n      <td>Effect on doping tests</td>\n      <td>Mandatory doping tests were being severely res...</td>\n      <td>163</td>\n      <td>2020 Summer Olympics\\nEffect on doping tests\\n...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: de77ba37-44bb-40b3-8360-d637a4dc3638
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"de77ba37-44bb-40b3-8360-d637a4dc3638\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7a7495e3-4d32-4e63-8e61-6f0435064238\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"53e69dbfc9bdbbbceb5c2b21703d5e273c29f14040fd9a6634604caa1d0396a5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e36e9eb8-b6ac-4d6a-8c00-16eca6cea118\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"37ff249c3e5c99e7e601e25d040626d3ec1872124757ebe04272156e33ff6db4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3219, \"end_char_idx\": 5081, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "## 2.2 Create questions based on the context\nUse davinci-instruct to generate a number of plausible questions relating to the Wikipedia section contents.\n\nNote: We have used temperature=0, but it may be beneficial to experiment with a higher temperature to get a higher diversity of questions.\n\n<span style=\"color:orange; font-weight:bold\">WARNING: This step will last a long time, and consume a lot of tokens, as it calls davinci-instruct for every section to generate a number of questions.</span>\n\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\ndef get_questions(context):\n    try:\n        response = client.chat.completions.create(model=\"davinci-instruct-beta-v3\",\n        prompt=f\"Write questions based on the text below\\n\\nText: {context}\\n\\nQuestions:\\n1.\",\n        temperature=0,\n        max_tokens=257,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n        stop=[\"\\n\\n\"])\n        return response.choices[0].text\n    except:\n        return \"\"\n\n\ndf['questions']= df.context.apply(get_questions)\ndf['questions'] = \"1.\" + df.questions\nprint(df[['questions']].values[0][0])\n```\n\n    1. What is the 2020 Summer Olympics?\n    2. When did the 2020 Summer Olympics take place?\n    3. Who won the most medals at the 2020 Summer Olympics?\n    4. Who won the most gold medals at the 2020 Summer Olympics?\n    5. Who won the most medals at the 2020 Summer Olympics?\n\n\nThe prompt is designed to generate a number of questions. Example questions above were generated based on the summary section of the 2020 Summer Olympics page.\n\nWe can observe that the questions 3 and 5 above repeat. Sometimes the generated questions could be ambiguous without the context. We will show that even despite these limitations we can create a successful model."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e36e9eb8-b6ac-4d6a-8c00-16eca6cea118
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e36e9eb8-b6ac-4d6a-8c00-16eca6cea118\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"de77ba37-44bb-40b3-8360-d637a4dc3638\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"99d66954bc4ef03b3cf6ff650655cf1f756b3a497c2103ca67af3fe54bbe8d6a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8ee49fc9-b7d9-4cdb-afae-cacd3f9964f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0726c13f839c826ae16182175605a6a17f0059ca5f8eeac18e89949dfc481ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3721, \"end_char_idx\": 8249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\ndef get_questions(context):\n    try:\n        response = client.chat.completions.create(model=\"davinci-instruct-beta-v3\",\n        prompt=f\"Write questions based on the text below\\n\\nText: {context}\\n\\nQuestions:\\n1.\",\n        temperature=0,\n        max_tokens=257,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n        stop=[\"\\n\\n\"])\n        return response.choices[0].text\n    except:\n        return \"\"\n\n\ndf['questions']= df.context.apply(get_questions)\ndf['questions'] = \"1.\" + df.questions\nprint(df[['questions']].values[0][0])\n```\n\n    1. What is the 2020 Summer Olympics?\n    2. When did the 2020 Summer Olympics take place?\n    3. Who won the most medals at the 2020 Summer Olympics?\n    4. Who won the most gold medals at the 2020 Summer Olympics?\n    5. Who won the most medals at the 2020 Summer Olympics?\n\n\nThe prompt is designed to generate a number of questions. Example questions above were generated based on the summary section of the 2020 Summer Olympics page.\n\nWe can observe that the questions 3 and 5 above repeat. Sometimes the generated questions could be ambiguous without the context. We will show that even despite these limitations we can create a successful model.\n\n\n```python\nprint(df.content.values[0])\n```\n\n    The 2020 Summer Olympics (Japanese: 2020\u5e74\u590f\u5b63\u30aa\u30ea\u30f3\u30d4\u30c3\u30af, Hepburn: Nisen Nij\u016b-nen Kaki Orinpikku), officially the Games of the XXXII Olympiad (\u7b2c\u4e09\u5341\u4e8c\u56de\u30aa\u30ea\u30f3\u30d4\u30c3\u30af\u7af6\u6280\u5927\u4f1a, Dai Sanj\u016bni-kai Orinpikku Ky\u014dgi Taikai) and branded as Tokyo 2020 (\u6771\u4eac2020, T\u014dky\u014d Nii Zero Nii Zero), was an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan, with some preliminary events that began on 21 July.\n    Tokyo was selected as the host city during the 125th IOC Session in Buenos Aires, Argentina, on 7 September 2013. Originally scheduled to take place from 24 July to 9 August 2020, the event was postponed to 2021 in March 2020 as a result of the COVID-19 pandemic, the first such instance in the history of the Olympic Games (previous games had been cancelled but not rescheduled). However, the event retained the Tokyo 2020 name for marketing and branding purposes. It was largely held behind closed doors with no public spectators permitted due to the declaration of a state of emergency in the Greater Tokyo Area in response to the pandemic. The Summer Paralympics were held between 24 August and 5 September 2021, 16 days after the completion of the Olympics.The 2020 Games were the fourth Olympic Games to be held in Japan, following the Tokyo 1964 (Summer), Sapporo 1972 (Winter) and Nagano 1998 (Winter) games. Tokyo is the first city in Asia to hold the Summer Games twice. The 2020 Games were the second of three consecutive Olympics to be held in East Asia, following the 2018 Winter Olympics in Pyeongchang, South Korea and preceding the 2022 Winter Olympics in Beijing, China.\n    New events were introduced in existing sports for 2020, including 3x3 basketball, freestyle BMX and mixed gender team events in a number of existing sports, as well as the return of madison cycling for men and an introduction of the same event for women. New IOC policies also allowed the host organizing committee to add new sports to the Olympic program for just one Games. The disciplines added by the Japanese Olympic Committee were baseball and softball, karate, sport climbing, surfing and skateboarding, the last four of which made their Olympic debuts, and the last three of which will remain on the Olympic program.The United States topped the medal count by both total golds (39) and total medals (113), with China finishing second by both respects (38 and 88). Host nation Japan finished third, setting a record for the most gold medals and total medals ever won by their delegation at an Olympic Games with 27 and 58. Great Britain finished fourth, with a total of 22 gold and 65 medals, becoming the first nation at the Summer Olympics to increase or equal their total medals won in the two Games subsequent to hosting them. The Russian delegation competing as the ROC (not to be confused with the Republic of China (Taiwan) which competed as Chinese Taipei, not ROC) finished fifth with 20 gold medals and third in the overall medal count, with 71 medals. Bermuda, the Philippines and Qatar won their first-ever Olympic gold medals. Burkina Faso, San Marino and Turkmenistan won their first-ever Olympic medals."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8ee49fc9-b7d9-4cdb-afae-cacd3f9964f5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8ee49fc9-b7d9-4cdb-afae-cacd3f9964f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e36e9eb8-b6ac-4d6a-8c00-16eca6cea118\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2433d66ea8e9f0de0d914f6fe1f01cee734c3949a80ea507bc87fb187f6420a2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7535479d-94e5-4bd3-b123-c901f825e156\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e2d82bcea0391beeace8fa174d127e5a91d9890e89be86df52aab2755179c96d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8252, \"end_char_idx\": 11280, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "## 2.3 Create answers based on the context\nUse davinci-instruct to answer the questions given the relevant Wikipedia section contents\n\nNote: We have used temperature=0, but it may be beneficial to experiment with a higher temperature to get a higher diversity of questions.\n\n<span style=\"color:orange\">**WARNING: This step will last a long time, and consume a lot of tokens, as it calls davinci-instruct for every section to answer all the questions.**</span>\n\n\n```python\ndef get_answers(row):\n    try:\n        response = client.chat.completions.create(\n            engine=\"davinci-instruct-beta-v3\",\n            prompt=f\"Write answer based on the text below\\n\\nText: {row.context}\\n\\nQuestions:\\n{row.questions}\\n\\nAnswers:\\n1.\",\n            temperature=0,\n            max_tokens=257,\n            top_p=1,\n            frequency_penalty=0,\n            presence_penalty=0\n        )\n        return response.choices[0].text\n    except Exception as e:\n        print (e)\n        return \"\"\n\n\ndf['answers']= df.apply(get_answers, axis=1)\ndf['answers'] = \"1.\" + df.answers\ndf = df.dropna().reset_index().drop('index',axis=1)\nprint(df[['answers']].values[0][0])\n```\n\n    1. The 2020 Summer Olympics is an international multi-sport event held from 23 July to 8 August 2021 in Tokyo, Japan.\n    2. The 2020 Summer Olympics took place from 23 July to 8 August 2021.\n    3. The United States topped the medal count by both total golds (39) and total medals (113), with China finishing second by both respects (38 and 88).\n    4. The United States topped the medal count by both total golds (39) and total medals (113), with China finishing second by both respects (38 and 88).\n    5. The United States topped the medal count by both total golds (39) and total medals (113), with China finishing second by both respects (38 and 88).\n\n\nThese are the answers to the questions above based on the context around the host city selection. \n\nWe can see that answers 3-5 contain the correct answer, but instead of answering the question directly, the answer is a verbatim extraction. Despite these occasional lower quality answers, we will show that the model can learn the task reasonably well, given a high number of examples.\n\n## 2.4 Save the Olympics Q&A dataset based on Wikipedia sections\nWe save the file for use in the [next notebook](olympics-3-train-qa.ipynb)\n\n\n```python\ndf.to_csv('olympics-data/olympics_qa.csv', index=False)\n```\n\n## 2.5 Search file (DEPRECATED)\nWe create a search file ([API reference](https://beta.openai.com/docs/api-reference/files/list)), which can be used to retrieve the relevant context when a question is asked.\n\n<span style=\"color:orange; font-weight:bold\">DEPRECATED: The /search endpoint is deprecated in favour of using embeddings. Embeddings are cheaper, faster and can support a better search experience. See <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering Guide</a> for a search implementation using the embeddings</span>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7535479d-94e5-4bd3-b123-c901f825e156
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7535479d-94e5-4bd3-b123-c901f825e156\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8ee49fc9-b7d9-4cdb-afae-cacd3f9964f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c71f9d3e71fef0d78365131954bfb36b8090d63bca544561e918c9a341f32b41\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4d2084be-3b7f-4148-9b0e-fb662081dd09\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b26160b2fc3dd4b93c22d0e38b68ab6b5ca5deab448fedc70b44d3852594faa9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10073, \"end_char_idx\": 12028, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "These are the answers to the questions above based on the context around the host city selection. \n\nWe can see that answers 3-5 contain the correct answer, but instead of answering the question directly, the answer is a verbatim extraction. Despite these occasional lower quality answers, we will show that the model can learn the task reasonably well, given a high number of examples.\n\n## 2.4 Save the Olympics Q&A dataset based on Wikipedia sections\nWe save the file for use in the [next notebook](olympics-3-train-qa.ipynb)\n\n\n```python\ndf.to_csv('olympics-data/olympics_qa.csv', index=False)\n```\n\n## 2.5 Search file (DEPRECATED)\nWe create a search file ([API reference](https://beta.openai.com/docs/api-reference/files/list)), which can be used to retrieve the relevant context when a question is asked.\n\n<span style=\"color:orange; font-weight:bold\">DEPRECATED: The /search endpoint is deprecated in favour of using embeddings. Embeddings are cheaper, faster and can support a better search experience. See <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering Guide</a> for a search implementation using the embeddings</span>\n\n\n\n```python\ndf = df[df.tokens<2000]\ndf[['context', 'tokens']].rename(columns={'context':'text','tokens':'metadata'}).to_json('olympics-data/olympics_search.jsonl', orient='records', lines=True)\n\nsearch_file = client.files.create(\n  file=open(\"olympics-data/olympics_search.jsonl\"),\n  purpose='search'\n)\nolympics_search_fileid = search_file['id']\n```\n\n## 2.6 Answer questions based on the context provided\n\nWe will use a simple implementation of the answers endpoint. This works by simply using the [/search endpoint](https://beta.openai.com/docs/api-reference/searches), which searches over an indexed file to obtain the relevant sections which can be included in the context, following by a question and answering prompt given a specified model."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4d2084be-3b7f-4148-9b0e-fb662081dd09
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4d2084be-3b7f-4148-9b0e-fb662081dd09\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7535479d-94e5-4bd3-b123-c901f825e156\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8555020ac8d606c917d2ea7806d3605c24b716c2b7fc6a4b7469c9b88e29a9b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dadee355-8961-4d32-ab1f-58f9f5fc96c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2f0267626d95ae41aff474e162dacff7692b380bdd626554cae4e525be23dca0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10602, \"end_char_idx\": 14076, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "```python\ndf.to_csv('olympics-data/olympics_qa.csv', index=False)\n```\n\n## 2.5 Search file (DEPRECATED)\nWe create a search file ([API reference](https://beta.openai.com/docs/api-reference/files/list)), which can be used to retrieve the relevant context when a question is asked.\n\n<span style=\"color:orange; font-weight:bold\">DEPRECATED: The /search endpoint is deprecated in favour of using embeddings. Embeddings are cheaper, faster and can support a better search experience. See <a href=\"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb\">Question Answering Guide</a> for a search implementation using the embeddings</span>\n\n\n\n```python\ndf = df[df.tokens<2000]\ndf[['context', 'tokens']].rename(columns={'context':'text','tokens':'metadata'}).to_json('olympics-data/olympics_search.jsonl', orient='records', lines=True)\n\nsearch_file = client.files.create(\n  file=open(\"olympics-data/olympics_search.jsonl\"),\n  purpose='search'\n)\nolympics_search_fileid = search_file['id']\n```\n\n## 2.6 Answer questions based on the context provided\n\nWe will use a simple implementation of the answers endpoint. This works by simply using the [/search endpoint](https://beta.openai.com/docs/api-reference/searches), which searches over an indexed file to obtain the relevant sections which can be included in the context, following by a question and answering prompt given a specified model.\n\n\n```python\nfrom answers_with_ft import create_context, answer_question\nprint(create_context(\"Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?\", olympics_search_fileid, max_len=400))\n```\n\n    Athletics at the 2020 Summer Olympics \u2013 Women's 4 \u00d7 100 metres relay\n    Summary\n    \n    The women's 4 \u00d7 100 metres relay event at the 2020 Summer Olympics took place on 5 and 6 August 2021 at the Japan National Stadium. There were 16 competing relay teams, with each team having 5 members from which 4 were selected in each round.\n    \n    ###\n    \n    Athletics at the 2020 Summer Olympics \u2013 Men's 4 \u00d7 100 metres relay\n    Qualification\n    \n    National Olympic Committees (NOCs) could qualify one relay team in one of three following ways:\n    The top 8 NOCs at the 2019 World Athletics Championships qualified a relay team.\n    The top 8 NOCs at the 2021 World Athletics Relays qualified a relay team.\n    Where an NOC placed in the top 8 at both the 2019 World Championships and the 2021 World Relays, the quota place was allocated to the world top list as of 29 June 2021. In this case, 4 teams did so, so there are 4 places available through the world rankings.A total of five athletes may be entered for a relay team. Should a NOC have also entered individual athletes in the corresponding individual event (100 m), the entered individual athletes must be included in the total of five (5) athletes entered for the relay event. In addition of five, NOCs can nominate a maximum of one alternate athlete for each team.\n    The qualifying period was originally from 1 May 2019 to 29 June 2020. Due to the COVID-19 pandemic, the period was suspended from 6 April 2020 to 30 November 2020, with the end date extended to 29 June 2021. The qualifying time standards could be obtained in various meets during the given period that have the approval of the IAAF. Both indoor and outdoor meets are eligible. The most recent Area Championships may be counted in the ranking, even if not during the qualifying period."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dadee355-8961-4d32-ab1f-58f9f5fc96c7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dadee355-8961-4d32-ab1f-58f9f5fc96c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4d2084be-3b7f-4148-9b0e-fb662081dd09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1ab8653f9e1ee77922b32ce059d6d440b703ff65d680d37bdd17e8f50ac4ab9a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8182cf38-cc3c-465b-b6c7-b33548246614\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e162a1ea5a2e13a63d47a7a226af8464a6db998ee2dfe66f4371b7ccdf6096d5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14080, \"end_char_idx\": 15448, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "```python\nanswer_question(olympics_search_fileid, \"davinci-instruct-beta-v3\", \n            \"Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?\")\n```\n\n\n\n\n    ' Japan National Stadium'\n\n\n\nAfter we fine-tune the model for Q&A we'll be able to use it instead of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), to obtain better answers when the question can't be answered based on the context. We see a downside of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), which always attempts to answer the question, regardless of the relevant context being present or not. (Note the second question is asking about a future event, set in 2024.)\n\n\n```python\nanswer_question(olympics_search_fileid, \"davinci-instruct-beta-v3\", \n            \"Where did women's 4 x 100 metres relay event take place during the 2048 Summer Olympics?\", max_len=1000)\n```\n\n\n\n\n    ' Japan National Stadium'\n\n\n\nWe can see that davinci has a tendency to answer the question, even if the question can't be answered given the context provided. Note the question asked regarding 2048 Summer Olympics, which didn't happen yet, and the retrieved content has only returned results for 2020.\n\n## 2.7 (Optional) Investigation into how likely the search endpoint is to return the relevant context"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8182cf38-cc3c-465b-b6c7-b33548246614
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8182cf38-cc3c-465b-b6c7-b33548246614\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dadee355-8961-4d32-ab1f-58f9f5fc96c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34e9028a357a41f9cca8c74390d551b1f8c2c2d81e3734a7f7e98b17cc2b7a4e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82eff7ba-3f2d-4630-b592-4bfb98a37a33\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c69dba5798852c1c363dc15b6b742c21b38c995a565652d205b7fd4e889cb418\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14080, \"end_char_idx\": 17343, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "```python\nanswer_question(olympics_search_fileid, \"davinci-instruct-beta-v3\", \n            \"Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?\")\n```\n\n\n\n\n    ' Japan National Stadium'\n\n\n\nAfter we fine-tune the model for Q&A we'll be able to use it instead of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), to obtain better answers when the question can't be answered based on the context. We see a downside of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), which always attempts to answer the question, regardless of the relevant context being present or not. (Note the second question is asking about a future event, set in 2024.)\n\n\n```python\nanswer_question(olympics_search_fileid, \"davinci-instruct-beta-v3\", \n            \"Where did women's 4 x 100 metres relay event take place during the 2048 Summer Olympics?\", max_len=1000)\n```\n\n\n\n\n    ' Japan National Stadium'\n\n\n\nWe can see that davinci has a tendency to answer the question, even if the question can't be answered given the context provided. Note the question asked regarding 2048 Summer Olympics, which didn't happen yet, and the retrieved content has only returned results for 2020.\n\n## 2.7 (Optional) Investigation into how likely the search endpoint is to return the relevant context\n\n\n```python\ndef check_context(title, heading, question, max_len=1800, search_model='ada', max_rerank=10):\n    \"\"\"\n    Evaluate the performance of the search model in retrieving the correct context\n\n    Parameters\n    ----------\n    title: str\n        The title of the Wikipedia page\n    heading: str\n        The heading of the Wikipedia section\n    qusetion: str\n        The question\n    max_len: int\n        The maximum length of the context\n    search_model: str\n        The search model to use - `ada` is most cost effective\n    max_rerank: int\n        The maximum number of reranking documents to use the search model on\n\n    Returns\n    -------\n    rank: int\n        The rank of the correct context\n    token_length: int\n        The number of tokens needed to obtain the correct context\n    \"\"\"\n    \n    try:\n        # TODO: openai.Engine(search_model) is deprecated\n        results = openai.Engine(search_model).search(\n            search_model=search_model, \n            query=question, \n            max_rerank=max_rerank,\n            file=olympics_search_fileid,\n            return_metadata=True\n        )\n        index=-1\n        returns = []\n        cur_len = 0\n        for result in results['data']:\n            cur_len += int(result['metadata']) + 4 # we add 4 tokens for the separator `\\n\\n###\\n\\n`\n            if cur_len > max_len:\n                break\n            returns.append(result['text'])\n            res = result['text'].split('\\n')\n            if res[0] == title and res[1] == heading:\n                index = len(returns) - 1\n                break\n        return index, cur_len\n    except Exception as e:\n        #print (e)\n        return []\nprint(check_context(\"Athletics at the 2020 Summer Olympics \u2013 Women's 4 \u00d7 100 metres relay\", \"Summary\", \"Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?\", max_len=10000))\n```\n\n    (0, 58)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 82eff7ba-3f2d-4630-b592-4bfb98a37a33
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"82eff7ba-3f2d-4630-b592-4bfb98a37a33\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8182cf38-cc3c-465b-b6c7-b33548246614\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0f42ffd7ea6c64a3678516d43a461551c7fd43d55eaaa817644db99ad72bb91a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"36c89705-3681-4389-ac6a-bcf6f05c00c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"068a6b7ab450d88ef40c83ff16b16f5c60058205e74bc92522d5612de1a2dfa8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17346, \"end_char_idx\": 20158, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "We utilize the generated questions based on context to estimate how often we can retrieve the original context. These questions are noisy, so this is not a perfect estimate.\n\nOur questions and answers are prefixed with numbered bullet points, however due to the way they were generated, they are missing the first number, hence we add \"1.\" to the list of questions (and answers).\n\nWe calculate the rank of the section retrieved using ada search, and the number of tokens in the context needed to retrieve the relevant section in full.\n\n\n```python\nada_results = df.apply(lambda x: [\n                    check_context( x.title, \n                                   x.heading, \n                                   q[3:],     # remove the number prefix\n                                   max_len=1000000, # set a large number to get the full context \n                                   search_model='ada', \n                                   max_rerank=200,\n                                 ) \n                    for q in (x.questions).split('\\n') # split the questions\n                    if len(q) >10 # remove the empty questions\n                ], axis=1)\nada_results.head()\n```\n\n\n\n\n    0    [(132, 27104), (-1, 22939), (8, 2151), (2, 121...\n    1    [(4, 1737), (0, 130), (8, 744), (96, 17208), (...\n    2          [(0, 373), (0, 373), (-1, 40610), (1, 570)]\n    3            [(0, 302), (0, 302), (5, 968), (8, 1425)]\n    4                      [(0, 167), (0, 167), (2, 1442)]\n    Name: ada, dtype: object\n\n\n\n\n```python\nout = pd.concat([ada_results], axis=1)\nout.columns = ['ada']\nout.to_csv('olympics-data/search_engine_results.csv')\n```\n\n\n```python\ndef expand_lists(out):\n    \"\"\"\n    Expand a pandas series containing lists into a series, where each list element becomes a value on its own\n\n    Input is a row per paragraph, which has multiple questions\n    Output is a row per question\n    \"\"\"\n    cols = [pd.DataFrame(out[name].tolist()).stack().reset_index(level=1, drop=True).rename(name) for name in out.columns] \n    return pd.concat(cols, axis=1)\n\nout_expanded = expand_lists(out)\nout_expanded['rank'] = out_expanded.ada.apply(lambda x: x[0] if x != [] else -2)\nout_expanded['tokens'] = out_expanded.ada.apply(lambda x: x[1] if x != [] else -2)\n\n```\n\n\n```python\nwithin_2k = (out_expanded.tokens < 2000).mean()\nprint(f\"{within_2k*100:.1f}% of relevant paragraphs are retrieved within the first 2k tokens\")\n```\n\n    74.3% of relevant paragraphs are retrieved within the first 2k tokens\n\n\nThe relevant context can be obtained 74% of the time on this dataset\n\n\n```python\noutside_200 = (out_expanded['rank'] == -1).mean()\nprint(f\"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results\")\n```\n\n    7.4% of relevant paragraphs are not retrieved within the first 200 results"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 36c89705-3681-4389-ac6a-bcf6f05c00c4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"36c89705-3681-4389-ac6a-bcf6f05c00c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82eff7ba-3f2d-4630-b592-4bfb98a37a33\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"af7b94890227ce0e7458929cbf91f39c263883e5743c188f038663591a330a54\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"751fd9c6-40cb-445a-9af3-8b9d73b68ae9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"062d44d0693558cf08d89eb8e656f1b0c99a05740d3b694619f9f64e1b71fe6e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18856, \"end_char_idx\": 21286, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "```python\nout = pd.concat([ada_results], axis=1)\nout.columns = ['ada']\nout.to_csv('olympics-data/search_engine_results.csv')\n```\n\n\n```python\ndef expand_lists(out):\n    \"\"\"\n    Expand a pandas series containing lists into a series, where each list element becomes a value on its own\n\n    Input is a row per paragraph, which has multiple questions\n    Output is a row per question\n    \"\"\"\n    cols = [pd.DataFrame(out[name].tolist()).stack().reset_index(level=1, drop=True).rename(name) for name in out.columns] \n    return pd.concat(cols, axis=1)\n\nout_expanded = expand_lists(out)\nout_expanded['rank'] = out_expanded.ada.apply(lambda x: x[0] if x != [] else -2)\nout_expanded['tokens'] = out_expanded.ada.apply(lambda x: x[1] if x != [] else -2)\n\n```\n\n\n```python\nwithin_2k = (out_expanded.tokens < 2000).mean()\nprint(f\"{within_2k*100:.1f}% of relevant paragraphs are retrieved within the first 2k tokens\")\n```\n\n    74.3% of relevant paragraphs are retrieved within the first 2k tokens\n\n\nThe relevant context can be obtained 74% of the time on this dataset\n\n\n```python\noutside_200 = (out_expanded['rank'] == -1).mean()\nprint(f\"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results\")\n```\n\n    7.4% of relevant paragraphs are not retrieved within the first 200 results\n\n\n7.4% of the time, this is due to the keyword search part of the search algorithm not retrieving the relevant context within the first 200 results.\n18.3% of the time this is due to the semantic search not placing the relevant context within the first 2000 tokens.\n\n\n```python\nimport matplotlib.pyplot as plt\n\n# plot a histogram, and add axis descriptions and title\nout_expanded[(out_expanded['rank'] >=0)&(out_expanded['rank'] <30)]['rank'].hist(bins=29)\nplt.xlabel('rank')\nplt.ylabel('count')\nplt.title('Histogram of ranks of retrieved paragraphs')\nplt.show()\n```\n\n\n    \n![png](output_31_0.png)\n    \n\n\n\n```python\nout_expanded[(out_expanded.tokens>=0)&(out_expanded.tokens < 2000)]['tokens'].hist(bins=29)\nplt.xlabel('tokens')\nplt.ylabel('count')\nplt.title('Histogram of the number of minimum tokens needed')\nplt.show()\n```\n\n\n    \n![png](output_32_0.png)\n    \n\n\nWe can observe that the context is most likely to be returned as one of the first results, and most likely to be returned within the first 200-500 tokens.\n\n\n```python\n# normalized value_counts\nout_expanded['rank'].value_counts(normalize=True).sort_index()[:13]\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 751fd9c6-40cb-445a-9af3-8b9d73b68ae9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"751fd9c6-40cb-445a-9af3-8b9d73b68ae9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"10552d4e-5cfb-45ba-967c-a752023a295f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cf0802646edd71df78869e6150069a0828e793d7919f3231dd5773cf3650c56f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"36c89705-3681-4389-ac6a-bcf6f05c00c4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb\", \"title\": \"2. Creating a synthetic Q&A dataset\", \"tokens\": 5416, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bf937d28507052d9d29c872087f451275ce9ea8af2ec0b9e20b25f63f1aad513\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19841, \"end_char_idx\": 21708, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "ref_doc_id": "10552d4e-5cfb-45ba-967c-a752023a295f", "chroma:document": "The relevant context can be obtained 74% of the time on this dataset\n\n\n```python\noutside_200 = (out_expanded['rank'] == -1).mean()\nprint(f\"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results\")\n```\n\n    7.4% of relevant paragraphs are not retrieved within the first 200 results\n\n\n7.4% of the time, this is due to the keyword search part of the search algorithm not retrieving the relevant context within the first 200 results.\n18.3% of the time this is due to the semantic search not placing the relevant context within the first 2000 tokens.\n\n\n```python\nimport matplotlib.pyplot as plt\n\n# plot a histogram, and add axis descriptions and title\nout_expanded[(out_expanded['rank'] >=0)&(out_expanded['rank'] <30)]['rank'].hist(bins=29)\nplt.xlabel('rank')\nplt.ylabel('count')\nplt.title('Histogram of ranks of retrieved paragraphs')\nplt.show()\n```\n\n\n    \n![png](output_31_0.png)\n    \n\n\n\n```python\nout_expanded[(out_expanded.tokens>=0)&(out_expanded.tokens < 2000)]['tokens'].hist(bins=29)\nplt.xlabel('tokens')\nplt.ylabel('count')\nplt.title('Histogram of the number of minimum tokens needed')\nplt.show()\n```\n\n\n    \n![png](output_32_0.png)\n    \n\n\nWe can observe that the context is most likely to be returned as one of the first results, and most likely to be returned within the first 200-500 tokens.\n\n\n```python\n# normalized value_counts\nout_expanded['rank'].value_counts(normalize=True).sort_index()[:13]\n```\n\n\n\n\n    -2     0.000063\n    -1     0.074428\n     0     0.453420\n     1     0.089515\n     2     0.047146\n     3     0.032437\n     4     0.024139\n     5     0.019676\n     6     0.015967\n     7     0.013452\n     8     0.011189\n     9     0.009869\n     10    0.009178\n    Name: rank, dtype: float64\n\n\n\nprobabilities of the relevant context being returned at each rank. (-2 means a processing error, -1 means the rank is >200)"}


