# embedding_metadata

key: _node_content
string_value:
{"id_": "3aafb590-3fd2-4955-a489-07b4d2741bfc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55c73e53-7d41-4365-94ce-fe44a8e67b78", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3eddd42036d3660c4791c5bddcb312e1ecc9d2a7c062cad41d86fb101ecb1142", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c54f886f-d7e3-436b-8d17-689e71419006", "node_type": "1", "metadata": {}, "hash": "7a71d7dd56e8801497d666ea8787e657c7d9207cc7e6c539915a061aa92bfa56", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16852, "end_char_idx": 19092, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
                    }
                },
                "required": ["coordinates"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "reject_request",
            "description": "Use this function if the request is not possible.",
            "parameters": {"type": "object", "properties": {}},
        },
    },
]
```

For starters, let's see how function calling performs with some straight forward feasible prompts, and then couple of obviously impossible requests which call the 'reject_request' function.



```python
straightforward_prompts_to_expected = {
    "Land the drone at the home base": "land_drone",
    "Take off the drone to 50 meters": "takeoff_drone",
    "Change speed to 15 kilometers per hour": "set_drone_speed",
    "Turn into an elephant!": "reject_request",
    "Move the drone forward by 10 meters": "control_drone_movement",
    "I want the LED display to blink in red": "configure_led_display",
    "Can you take a photo?": "control_camera",
    "Can you detect obstacles?": "set_obstacle_avoidance",
    "Can you dance for me?": "reject_request",
    "Can you follow me?": "set_follow_me_mode",
}
```


```python
# Evaluate the model with the given prompts
eval(
    model="gpt-3.5-turbo",
    system_prompt=DRONE_SYSTEM_PROMPT,
    function_list=function_list,
    prompts_to_expected_tool_name=straightforward_prompts_to_expected,
)
```


<style type="text/css">
#T_b01a0_row0_col0, #T_b01a0_row0_col1, #T_b01a0_row0_col2, #T_b01a0_row0_col3, #T_b01a0_row1_col0, #T_b01a0_row1_col1, #T_b01a0_row1_col2, #T_b01a0_row1_col3, #T_b01a0_row2_col0, #T_b01a0_row2_col1, #T_b01a0_row2_col2, #T_b01a0_row2_col3, #T_b01a0_row3_col0, #T_b01a0_row3_col1, #T_b01a0_row3_col2, #T_b01a0_row3_col3, #T_b01a0_row4_col0, #T_b01a0_row4_col1, #T_b01a0_row4_col2, #T_b01a0_row4_col3, #T_b01a0_row5_col0, #T_b01a0_row5_col1, #T_b01a0_row5_col2, #T_b01a0_row5_col3, #T_b01a0_row6_col0, #T_b01a0_row6_col1, #T_b01a0_row6_col2, #T_b01a0_row6_col3, #T_b01a0_row7_col0, #T_b01a0_row7_col1, #T_b01a0_row7_col2, #T_b01a0_row7_col3, #T_b01a0_row8_col0, #T_b01a0_row8_col1, #T_b01a0_row8_col2, #T_b01a0_row8_col3, #T_b01a0_row9_col0, #T_b01a0_row9_col1, #T_b01a0_row9_col2,


key: _node_content
string_value:
{"id_": "c54f886f-d7e3-436b-8d17-689e71419006", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3aafb590-3fd2-4955-a489-07b4d2741bfc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0e905e60150b1f90fb85b4fcb35c47f4946c3070ba3eae1d3ec9d52da894368f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23a92b94-6a05-4bd5-838d-c271dc303179", "node_type": "1", "metadata": {}, "hash": "8fb98215835a62423081338fd157865f2ac4aec2f2398a4cca63e43bbe0bcc94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18373, "end_char_idx": 19977, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#T_b01a0_row0_col3, #T_b01a0_row1_col0, #T_b01a0_row1_col1, #T_b01a0_row1_col2, #T_b01a0_row1_col3, #T_b01a0_row2_col0, #T_b01a0_row2_col1, #T_b01a0_row2_col2, #T_b01a0_row2_col3, #T_b01a0_row3_col0, #T_b01a0_row3_col1, #T_b01a0_row3_col2, #T_b01a0_row3_col3, #T_b01a0_row4_col0, #T_b01a0_row4_col1, #T_b01a0_row4_col2, #T_b01a0_row4_col3, #T_b01a0_row5_col0, #T_b01a0_row5_col1, #T_b01a0_row5_col2, #T_b01a0_row5_col3, #T_b01a0_row6_col0, #T_b01a0_row6_col1, #T_b01a0_row6_col2, #T_b01a0_row6_col3, #T_b01a0_row7_col0, #T_b01a0_row7_col1, #T_b01a0_row7_col2, #T_b01a0_row7_col3, #T_b01a0_row8_col0, #T_b01a0_row8_col1, #T_b01a0_row8_col2, #T_b01a0_row8_col3, #T_b01a0_row9_col0, #T_b01a0_row9_col1, #T_b01a0_row9_col2, #T_b01a0_row9_col3 {
  background-color: white;
  color: black;
}
</style>
<table id="T_b01a0">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_b01a0_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_b01a0_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_b01a0_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_b01a0_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_b01a0_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_b01a0_row0_col0" class="data row0 col0" >Land the drone at the home base</td>
      <td id="T_b01a0_row0_col1" class="data row0 col1" >land_drone</td>
      <td id="T_b01a0_row0_col2" class="data row0 col2" >land_drone</td>
      <td id="T_b01a0_row0_col3" class="data row0


key: _node_content
string_value:
{"id_": "23a92b94-6a05-4bd5-838d-c271dc303179", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c54f886f-d7e3-436b-8d17-689e71419006", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "cdad9d878fe1f2d828498f09658d2c05d66ac905dfb98c7c96384e7ee1f377e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ccca5055-c53f-46c1-8f39-486ac6c42080", "node_type": "1", "metadata": {}, "hash": "1911d06797c0c5a89e9f69dd933499ad41e3c337eeb6fb78ddc4e0d6e741bb57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19053, "end_char_idx": 20806, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#T_b01a0_row9_col1, #T_b01a0_row9_col2, #T_b01a0_row9_col3 {
  background-color: white;
  color: black;
}
</style>
<table id="T_b01a0">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_b01a0_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_b01a0_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_b01a0_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_b01a0_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_b01a0_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_b01a0_row0_col0" class="data row0 col0" >Land the drone at the home base</td>
      <td id="T_b01a0_row0_col1" class="data row0 col1" >land_drone</td>
      <td id="T_b01a0_row0_col2" class="data row0 col2" >land_drone</td>
      <td id="T_b01a0_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_b01a0_row1_col0" class="data row1 col0" >Take off the drone to 50 meters</td>
      <td id="T_b01a0_row1_col1" class="data row1 col1" >takeoff_drone</td>
      <td id="T_b01a0_row1_col2" class="data row1 col2" >takeoff_drone</td>
      <td id="T_b01a0_row1_col3" class="data row1 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_b01a0_row2_col0" class="data row2 col0" >Change speed to 15 kilometers per hour</td>
      <td id="T_b01a0_row2_col1" class="data row2 col1" >set_drone_speed</td>
      <td id="T_b01a0_row2_col2" class="data row2 col2" >set_drone_speed</td>
      <td id="T_b01a0_row2_col3" class="data row2 col3"


key: _node_content
string_value:
{"id_": "ccca5055-c53f-46c1-8f39-486ac6c42080", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23a92b94-6a05-4bd5-838d-c271dc303179", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "bab7c2ebac3c36b709d78704e294e5693d158d755b71351abec8e64792707308", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd1f6d22-95c5-46e6-ad97-34607a1f1622", "node_type": "1", "metadata": {}, "hash": "5df8c3895d9755ade5293d712eac46cbf30442e3ed3ab1351f82c3ae31a3e69e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19934, "end_char_idx": 21643, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_b01a0_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_b01a0_row1_col0" class="data row1 col0" >Take off the drone to 50 meters</td>
      <td id="T_b01a0_row1_col1" class="data row1 col1" >takeoff_drone</td>
      <td id="T_b01a0_row1_col2" class="data row1 col2" >takeoff_drone</td>
      <td id="T_b01a0_row1_col3" class="data row1 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_b01a0_row2_col0" class="data row2 col0" >Change speed to 15 kilometers per hour</td>
      <td id="T_b01a0_row2_col1" class="data row2 col1" >set_drone_speed</td>
      <td id="T_b01a0_row2_col2" class="data row2 col2" >set_drone_speed</td>
      <td id="T_b01a0_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_b01a0_row3_col0" class="data row3 col0" >Turn into an elephant!</td>
      <td id="T_b01a0_row3_col1" class="data row3 col1" >reject_request</td>
      <td id="T_b01a0_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_b01a0_row3_col3" class="data row3 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_b01a0_row4_col0" class="data row4 col0" >Move the drone forward by 10 meters</td>
      <td id="T_b01a0_row4_col1" class="data row4 col1" >control_drone_movement</td>
      <td id="T_b01a0_row4_col2" class="data row4 col2" >control_drone_movement</td>
      <td id="T_b01a0_row4_col3" class="data row4 col3" >Yes</td>


key: _node_content
string_value:
{"id_": "cd1f6d22-95c5-46e6-ad97-34607a1f1622", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ccca5055-c53f-46c1-8f39-486ac6c42080", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "64ed8567f391da2cc3f5a96fed4906963ed49bddd2f0060903a811a342073eba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f4d7091-5069-4319-95d3-6c167d9cbb98", "node_type": "1", "metadata": {}, "hash": "aa0fc8b144e319bf640d4ff196dc7dfbfa42b1a31f654da7ddc681cf69e0d940", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20757, "end_char_idx": 22489, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_b01a0_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_b01a0_row3_col0" class="data row3 col0" >Turn into an elephant!</td>
      <td id="T_b01a0_row3_col1" class="data row3 col1" >reject_request</td>
      <td id="T_b01a0_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_b01a0_row3_col3" class="data row3 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_b01a0_row4_col0" class="data row4 col0" >Move the drone forward by 10 meters</td>
      <td id="T_b01a0_row4_col1" class="data row4 col1" >control_drone_movement</td>
      <td id="T_b01a0_row4_col2" class="data row4 col2" >control_drone_movement</td>
      <td id="T_b01a0_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_b01a0_row5_col0" class="data row5 col0" >I want the LED display to blink in red</td>
      <td id="T_b01a0_row5_col1" class="data row5 col1" >configure_led_display</td>
      <td id="T_b01a0_row5_col2" class="data row5 col2" >configure_led_display</td>
      <td id="T_b01a0_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_b01a0_row6_col0" class="data row6 col0" >Can you take a photo?</td>
      <td id="T_b01a0_row6_col1" class="data row6 col1" >control_camera</td>
      <td id="T_b01a0_row6_col2" class="data row6 col2" >control_camera</td>
      <td id="T_b01a0_row6_col3" class="data row6 col3" >Yes</td>
    </tr>
    <tr>


key: _node_content
string_value:
{"id_": "9f4d7091-5069-4319-95d3-6c167d9cbb98", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd1f6d22-95c5-46e6-ad97-34607a1f1622", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9448de77e71fc01cb65b6b6bb8d759ffa13bc547b9930443b2462769a5464b11", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f093754-bfe1-4c97-9de4-8c45ec40b76a", "node_type": "1", "metadata": {}, "hash": "4ad821a896ad716de7bcc5fa03c178fe35bf57796cac05acbca1250db3a88c30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21611, "end_char_idx": 23340, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_b01a0_row5_col0" class="data row5 col0" >I want the LED display to blink in red</td>
      <td id="T_b01a0_row5_col1" class="data row5 col1" >configure_led_display</td>
      <td id="T_b01a0_row5_col2" class="data row5 col2" >configure_led_display</td>
      <td id="T_b01a0_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_b01a0_row6_col0" class="data row6 col0" >Can you take a photo?</td>
      <td id="T_b01a0_row6_col1" class="data row6 col1" >control_camera</td>
      <td id="T_b01a0_row6_col2" class="data row6 col2" >control_camera</td>
      <td id="T_b01a0_row6_col3" class="data row6 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_b01a0_row7_col0" class="data row7 col0" >Can you detect obstacles?</td>
      <td id="T_b01a0_row7_col1" class="data row7 col1" >set_obstacle_avoidance</td>
      <td id="T_b01a0_row7_col2" class="data row7 col2" >set_obstacle_avoidance</td>
      <td id="T_b01a0_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_b01a0_row8_col0" class="data row8 col0" >Can you dance for me?</td>
      <td id="T_b01a0_row8_col1" class="data row8 col1" >reject_request</td>
      <td id="T_b01a0_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_b01a0_row8_col3" class="data row8 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row9"


key: _node_content
string_value:
{"id_": "4f093754-bfe1-4c97-9de4-8c45ec40b76a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f4d7091-5069-4319-95d3-6c167d9cbb98", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "37f2692b99a575abbbc124a5fb9a51d2bcdc94d2678e2306bdcdd3d0995549e7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fd4bdfd-9855-4020-94d7-5b59104492a7", "node_type": "1", "metadata": {}, "hash": "ca0a30934681d4f01a9f388d711abaf886f2efd88cf5d7ca716aea6ed8d47c50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22475, "end_char_idx": 24085, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</tr>
    <tr>
      <th id="T_b01a0_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_b01a0_row7_col0" class="data row7 col0" >Can you detect obstacles?</td>
      <td id="T_b01a0_row7_col1" class="data row7 col1" >set_obstacle_avoidance</td>
      <td id="T_b01a0_row7_col2" class="data row7 col2" >set_obstacle_avoidance</td>
      <td id="T_b01a0_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_b01a0_row8_col0" class="data row8 col0" >Can you dance for me?</td>
      <td id="T_b01a0_row8_col1" class="data row8 col1" >reject_request</td>
      <td id="T_b01a0_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_b01a0_row8_col3" class="data row8 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_b01a0_row9_col0" class="data row9 col0" >Can you follow me?</td>
      <td id="T_b01a0_row9_col1" class="data row9 col1" >set_follow_me_mode</td>
      <td id="T_b01a0_row9_col2" class="data row9 col2" >set_follow_me_mode</td>
      <td id="T_b01a0_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 10 out of 10 (100.00%)
    Average latency per request: 826.81 ms
    Average tokens used per request: 796.20


Nice! The model performs quite well with these requests. Now let's try some more difficult requests: requests that are _almost_ feasible and are drone-related, but that the drone cannot actually do, and the pilot should reject.


key: _node_content
string_value:
{"id_": "2fd4bdfd-9855-4020-94d7-5b59104492a7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f093754-bfe1-4c97-9de4-8c45ec40b76a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "830bc68eacc5d54a21dfd282925633c3ec26fb1150c612af3bf9a258a0422de0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df574749-b27d-44b8-810c-d00d2a87035e", "node_type": "1", "metadata": {}, "hash": "5ae6de274178b7a35d3c1c613f406eeb967b80877c973ee33f493144861b36f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23016, "end_char_idx": 25255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="data row8 col0" >Can you dance for me?</td>
      <td id="T_b01a0_row8_col1" class="data row8 col1" >reject_request</td>
      <td id="T_b01a0_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_b01a0_row8_col3" class="data row8 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_b01a0_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_b01a0_row9_col0" class="data row9 col0" >Can you follow me?</td>
      <td id="T_b01a0_row9_col1" class="data row9 col1" >set_follow_me_mode</td>
      <td id="T_b01a0_row9_col2" class="data row9 col2" >set_follow_me_mode</td>
      <td id="T_b01a0_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 10 out of 10 (100.00%)
    Average latency per request: 826.81 ms
    Average tokens used per request: 796.20


Nice! The model performs quite well with these requests. Now let's try some more difficult requests: requests that are _almost_ feasible and are drone-related, but that the drone cannot actually do, and the pilot should reject.



```python
challenging_prompts_to_expected = {
    "Play pre-recorded audio message": "reject_request",
    "Initiate following on social media": "reject_request",
    "Scan environment for heat signatures": "reject_request",
    "Bump into obstacles": "reject_request",
    "Change drone's paint job color": "reject_request",
    "Coordinate with nearby drones": "reject_request",
    "Change speed to negative 120 km/h": "reject_request",
    "Detect a person": "reject_request",
    "Please enable night vision": "reject_request",
    "Report on humidity levels around you": "reject_request",
}
```


```python
# Evaluate the model with the challenging prompts
eval(
    model="gpt-3.5-turbo",
    function_list=function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)
```


<style type="text/css">
#T_99c20_row0_col0, #T_99c20_row0_col1, #T_99c20_row0_col2, #T_99c20_row0_col3, #T_99c20_row2_col0, #T_99c20_row2_col1, #T_99c20_row2_col2, #T_99c20_row2_col3, #T_99c20_row4_col0, #T_99c20_row4_col1, #T_99c20_row4_col2, #T_99c20_row4_col3, #T_99c20_row5_col0, #T_99c20_row5_col1, #T_99c20_row5_col2,


key: _node_content
string_value:
{"id_": "df574749-b27d-44b8-810c-d00d2a87035e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fd4bdfd-9855-4020-94d7-5b59104492a7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3df7c63faace321ab7be866f0f98d1c407f64eb69ac1de50c7f434a288a06928", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed00e2ad-1448-4e73-aa4e-dde8e5b3f290", "node_type": "1", "metadata": {}, "hash": "03af495ea6d16222415a155b5a22aa29d095af25d4a6dea5c466621c95e78345", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24089, "end_char_idx": 26085, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
challenging_prompts_to_expected = {
    "Play pre-recorded audio message": "reject_request",
    "Initiate following on social media": "reject_request",
    "Scan environment for heat signatures": "reject_request",
    "Bump into obstacles": "reject_request",
    "Change drone's paint job color": "reject_request",
    "Coordinate with nearby drones": "reject_request",
    "Change speed to negative 120 km/h": "reject_request",
    "Detect a person": "reject_request",
    "Please enable night vision": "reject_request",
    "Report on humidity levels around you": "reject_request",
}
```


```python
# Evaluate the model with the challenging prompts
eval(
    model="gpt-3.5-turbo",
    function_list=function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)
```


<style type="text/css">
#T_99c20_row0_col0, #T_99c20_row0_col1, #T_99c20_row0_col2, #T_99c20_row0_col3, #T_99c20_row2_col0, #T_99c20_row2_col1, #T_99c20_row2_col2, #T_99c20_row2_col3, #T_99c20_row4_col0, #T_99c20_row4_col1, #T_99c20_row4_col2, #T_99c20_row4_col3, #T_99c20_row5_col0, #T_99c20_row5_col1, #T_99c20_row5_col2, #T_99c20_row5_col3, #T_99c20_row7_col0, #T_99c20_row7_col1, #T_99c20_row7_col2, #T_99c20_row7_col3, #T_99c20_row9_col0, #T_99c20_row9_col1, #T_99c20_row9_col2, #T_99c20_row9_col3 {
  background-color: white;
  color: black;
}
#T_99c20_row1_col0, #T_99c20_row1_col1, #T_99c20_row1_col2, #T_99c20_row1_col3, #T_99c20_row3_col0, #T_99c20_row3_col1, #T_99c20_row3_col2, #T_99c20_row3_col3, #T_99c20_row6_col0, #T_99c20_row6_col1, #T_99c20_row6_col2, #T_99c20_row6_col3, #T_99c20_row8_col0, #T_99c20_row8_col1, #T_99c20_row8_col2, #T_99c20_row8_col3 {
  background-color: red;
  color: black;
}
</style>
<table id="T_99c20">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_99c20_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_99c20_level0_col1" class="col_heading level0 col1"


key: _node_content
string_value:
{"id_": "ed00e2ad-1448-4e73-aa4e-dde8e5b3f290", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df574749-b27d-44b8-810c-d00d2a87035e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0be580060dd94c481e535cc1b817b147f685ca188376764534150a84491a286e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d58bc4ce-182d-4d9e-9f1c-e22d1da7342c", "node_type": "1", "metadata": {}, "hash": "af1bc7b99b966e5828a08ad1bb375d0eb7b9fd40274c92e444b476e6af707589", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25256, "end_char_idx": 26935, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#T_99c20_row5_col3, #T_99c20_row7_col0, #T_99c20_row7_col1, #T_99c20_row7_col2, #T_99c20_row7_col3, #T_99c20_row9_col0, #T_99c20_row9_col1, #T_99c20_row9_col2, #T_99c20_row9_col3 {
  background-color: white;
  color: black;
}
#T_99c20_row1_col0, #T_99c20_row1_col1, #T_99c20_row1_col2, #T_99c20_row1_col3, #T_99c20_row3_col0, #T_99c20_row3_col1, #T_99c20_row3_col2, #T_99c20_row3_col3, #T_99c20_row6_col0, #T_99c20_row6_col1, #T_99c20_row6_col2, #T_99c20_row6_col3, #T_99c20_row8_col0, #T_99c20_row8_col1, #T_99c20_row8_col2, #T_99c20_row8_col3 {
  background-color: red;
  color: black;
}
</style>
<table id="T_99c20">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_99c20_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_99c20_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_99c20_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_99c20_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_99c20_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_99c20_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_99c20_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_99c20_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_99c20_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_99c20_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_99c20_row1_col1" class="data row1 col1"


key: _node_content
string_value:
{"id_": "d58bc4ce-182d-4d9e-9f1c-e22d1da7342c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed00e2ad-1448-4e73-aa4e-dde8e5b3f290", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1c088620d78fadaa48b71194f50880ef4a02d7087875c0b20524cb44d91eed09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f39ceed8-7f0f-4123-b92b-726226d75263", "node_type": "1", "metadata": {}, "hash": "2dab33f6d940bf817ce791cd8e35166c8f37ccccf1df387c4733152144a00ada", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26025, "end_char_idx": 27786, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<th id="T_99c20_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_99c20_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_99c20_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_99c20_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_99c20_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_99c20_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_99c20_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_99c20_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_99c20_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_99c20_row1_col1" class="data row1 col1" >set_follow_me_mode</td>
      <td id="T_99c20_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_99c20_row1_col3" class="data row1 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_99c20_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_99c20_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_99c20_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_99c20_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_99c20_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_99c20_row3_col1" class="data row3 col1" >set_obstacle_avoidance</td>
      <td


key: _node_content
string_value:
{"id_": "f39ceed8-7f0f-4123-b92b-726226d75263", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d58bc4ce-182d-4d9e-9f1c-e22d1da7342c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f126a03a521d0b429e6c963d53841c0d49a863cc1d17a08756ae4c768174e532", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "528543d3-08e9-4326-b7a3-ab89b2ad6b34", "node_type": "1", "metadata": {}, "hash": "533ba1fa91c68ed807c404625ba14c81fc56e53430eb4c8484e9bc13a3389901", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26886, "end_char_idx": 28633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_99c20_row1_col1" class="data row1 col1" >set_follow_me_mode</td>
      <td id="T_99c20_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_99c20_row1_col3" class="data row1 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_99c20_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_99c20_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_99c20_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_99c20_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_99c20_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_99c20_row3_col1" class="data row3 col1" >set_obstacle_avoidance</td>
      <td id="T_99c20_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_99c20_row3_col3" class="data row3 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_99c20_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_99c20_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_99c20_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_99c20_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_99c20_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_99c20_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_99c20_row5_col2" class="data


key: _node_content
string_value:
{"id_": "528543d3-08e9-4326-b7a3-ab89b2ad6b34", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f39ceed8-7f0f-4123-b92b-726226d75263", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e6e878b00445b96c184eef2e78f94a40e3f27be640d34515e2b376a8805c1621", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7eda091-8c38-4eb7-9d8f-97d425d7b2dc", "node_type": "1", "metadata": {}, "hash": "1eeb26238f8dec1ced8908ddf5e8c315638e008826587755e963360436c0b798", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27725, "end_char_idx": 29467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="data row3 col1" >set_obstacle_avoidance</td>
      <td id="T_99c20_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_99c20_row3_col3" class="data row3 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_99c20_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_99c20_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_99c20_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_99c20_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_99c20_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_99c20_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_99c20_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_99c20_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_99c20_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_99c20_row6_col1" class="data row6 col1" >set_drone_speed</td>
      <td id="T_99c20_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_99c20_row6_col3" class="data row6 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_99c20_row7_col0" class="data row7 col0" >Detect a person</td>
      <td id="T_99c20_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_99c20_row7_col2" class="data row7 col2" >reject_request</td>


key: _node_content
string_value:
{"id_": "e7eda091-8c38-4eb7-9d8f-97d425d7b2dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "528543d3-08e9-4326-b7a3-ab89b2ad6b34", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0a0b0465bd2f35d28f8ff47bdf14dd9f0d0d1a85a8b348b97aa8b50fe6756501", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0bdf8ab-7aec-4e56-8246-e011d7f708fe", "node_type": "1", "metadata": {}, "hash": "0dda9a47d6099a5e419e46c1004c5654aac18507d2f806789c136402ca2fa325", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28595, "end_char_idx": 30296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_99c20_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_99c20_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_99c20_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_99c20_row6_col1" class="data row6 col1" >set_drone_speed</td>
      <td id="T_99c20_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_99c20_row6_col3" class="data row6 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_99c20_row7_col0" class="data row7 col0" >Detect a person</td>
      <td id="T_99c20_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_99c20_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_99c20_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_99c20_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_99c20_row8_col1" class="data row8 col1" >set_drone_lighting</td>
      <td id="T_99c20_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_99c20_row8_col3" class="data row8 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_99c20_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_99c20_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_99c20_row9_col2" class="data row9 col2" >reject_request</td>
      <td


key: _node_content
string_value:
{"id_": "b0bdf8ab-7aec-4e56-8246-e011d7f708fe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7eda091-8c38-4eb7-9d8f-97d425d7b2dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9815868a96a82aae69991cf8602246cecee39fef0dd58a2c4da0392a64b12b1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "01dd1be5-88f5-4105-9b81-dde3657f5c30", "node_type": "1", "metadata": {}, "hash": "e929d6833418b0c3ac4cb65ebfb1329b88059ea2806617bd3fe5f6c182161f64", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29397, "end_char_idx": 31942, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_99c20_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_99c20_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_99c20_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_99c20_row8_col1" class="data row8 col1" >set_drone_lighting</td>
      <td id="T_99c20_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_99c20_row8_col3" class="data row8 col3" >No</td>
    </tr>
    <tr>
      <th id="T_99c20_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_99c20_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_99c20_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_99c20_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_99c20_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 6 out of 10 (60.00%)
    Average latency per request: 610.26 ms
    Average tokens used per request: 791.90


Now we run into some problems.
The model here should reject all of these requests, as they are impossible/conflicting/ambiguous given the functions, however instead the model calls functions that are somewhat related to the request, but incorrect. For example, the model sets follow_me_mode when asked to initiate following on social media.

<br>
In this simple case, more prompt engineering may resolve some of these issues, but for the purpose of this example we will demonstrate how fine tuning can be used to improve performance. Additionally, while this case is relatively straightforward, as the number of and complexity of the functions increases, fine tuning becomes more and more impactful.

Again, our goal here is to improve performance and use less tokens, so fine-tuning allows us to:

- Omit function and parameter descriptions: remove the description field from function and parameters
- Omit parameters: remove the entire properties field from the parameters object
- Omit function entirely: remove the entire function object from the functions array


# Generating synthetic data


### Helper functions


We want to generate every invocation of every function, so that we have
full coverage of all potential invocations to create synthetic data for. Then, we will use `gpt-4o` to come up with prompts that would call each invocation, and we will use that prompt - function invocation pair as training data.


key: _node_content
string_value:
{"id_": "01dd1be5-88f5-4105-9b81-dde3657f5c30", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0bdf8ab-7aec-4e56-8246-e011d7f708fe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7b9d86a6bd485c58ae4731ca24238ec3eb90b5dbb9028c9051d8f5e981b77ab8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c6e00e1b-cc88-4465-84ec-6b2a55ffd5b4", "node_type": "1", "metadata": {}, "hash": "edc60e07775f0fecace101e6faa2dc2387528bf4072d4b8fd85b8747821d105d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30189, "end_char_idx": 33378, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>reject_request</td>
      <td id="T_99c20_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_99c20_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 6 out of 10 (60.00%)
    Average latency per request: 610.26 ms
    Average tokens used per request: 791.90


Now we run into some problems.
The model here should reject all of these requests, as they are impossible/conflicting/ambiguous given the functions, however instead the model calls functions that are somewhat related to the request, but incorrect. For example, the model sets follow_me_mode when asked to initiate following on social media.

<br>
In this simple case, more prompt engineering may resolve some of these issues, but for the purpose of this example we will demonstrate how fine tuning can be used to improve performance. Additionally, while this case is relatively straightforward, as the number of and complexity of the functions increases, fine tuning becomes more and more impactful.

Again, our goal here is to improve performance and use less tokens, so fine-tuning allows us to:

- Omit function and parameter descriptions: remove the description field from function and parameters
- Omit parameters: remove the entire properties field from the parameters object
- Omit function entirely: remove the entire function object from the functions array


# Generating synthetic data


### Helper functions


We want to generate every invocation of every function, so that we have
full coverage of all potential invocations to create synthetic data for. Then, we will use `gpt-4o` to come up with prompts that would call each invocation, and we will use that prompt - function invocation pair as training data.


Generating every invocation for a function with fixed enums is more simple, but for a function such as
`control_gimbal` we need to set the `tilt` and `pan` integer values, so to generate those synthetic invocations we will first set a placeholder, and then later use `gpt-4o` to come up with reasonable values.



```python
placeholder_int = "fill_in_int"
placeholder_string = "fill_in_string"
```

The functions below take in all the functions from the function list, and look
at all the potential invocations of those functions given each function's parameters.
The functions also account for `required` parameters, so that all the invocations
are actually feasible.



```python
def generate_permutations(
    params: Dict[str, Dict[str, Any]]
) -> Generator[Dict[str, Any], None, None]:
    """
    Generates all possible permutations for given parameters.

    :param params: Parameter dictionary containing required and optional fields.
    :return: A generator yielding each permutation.
    """

    # Extract the required fields from the parameters
    required_fields = params.get("required", [])

    # Generate permutations for required fields
    required_permutations = generate_required_permutations(params, required_fields)

    # Generate optional permutations based on each required permutation
    for required_perm in required_permutations:
        yield from generate_optional_permutations(params, required_perm)


key: _node_content
string_value:
{"id_": "c6e00e1b-cc88-4465-84ec-6b2a55ffd5b4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "01dd1be5-88f5-4105-9b81-dde3657f5c30", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "28a7d8cd01a9bb929c44333276c81e8c192513728bffad745e487af561018c14", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24458b10-beb5-408f-8f7f-68754050e6f2", "node_type": "1", "metadata": {}, "hash": "0936691a0f4e7bbb6a8a0669b6e1810241cc3ae7a52d47f7672469ca732468ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31588, "end_char_idx": 35246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Generating synthetic data


### Helper functions


We want to generate every invocation of every function, so that we have
full coverage of all potential invocations to create synthetic data for. Then, we will use `gpt-4o` to come up with prompts that would call each invocation, and we will use that prompt - function invocation pair as training data.


Generating every invocation for a function with fixed enums is more simple, but for a function such as
`control_gimbal` we need to set the `tilt` and `pan` integer values, so to generate those synthetic invocations we will first set a placeholder, and then later use `gpt-4o` to come up with reasonable values.



```python
placeholder_int = "fill_in_int"
placeholder_string = "fill_in_string"
```

The functions below take in all the functions from the function list, and look
at all the potential invocations of those functions given each function's parameters.
The functions also account for `required` parameters, so that all the invocations
are actually feasible.



```python
def generate_permutations(
    params: Dict[str, Dict[str, Any]]
) -> Generator[Dict[str, Any], None, None]:
    """
    Generates all possible permutations for given parameters.

    :param params: Parameter dictionary containing required and optional fields.
    :return: A generator yielding each permutation.
    """

    # Extract the required fields from the parameters
    required_fields = params.get("required", [])

    # Generate permutations for required fields
    required_permutations = generate_required_permutations(params, required_fields)

    # Generate optional permutations based on each required permutation
    for required_perm in required_permutations:
        yield from generate_optional_permutations(params, required_perm)


def generate_required_permutations(
    params: Dict[str, Dict[str, Any]], required_fields: List[str]
) -> List[Dict[str, Any]]:
    """
    Generates permutations for the required fields.

    :param params: Parameter dictionary.
    :param required_fields: List of required fields.
    :return: A list of permutations for required fields.
    """

    # Get all possible values for each required field
    required_values = [get_possible_values(params, field) for field in required_fields]

    # Generate permutations from possible values
    return [
        dict(zip(required_fields, values))
        for values in itertools.product(*required_values)
    ]


def generate_optional_permutations(
    params: Dict[str, Dict[str, Any]], base_perm: Dict[str, Any]
) -> Generator[Dict[str, Any], None, None]:
    """
    Generates permutations for optional fields based on a base permutation.

    :param params: Parameter dictionary.
    :param base_perm: Base permutation dictionary.
    :return: A generator yielding each permutation for optional fields.
    """

    # Determine the fields that are optional by subtracting the base permutation's fields from all properties
    optional_fields = set(params["properties"]) - set(base_perm)

    # Iterate through all combinations of optional fields
    for field_subset in itertools.chain.from_iterable(
        itertools.combinations(optional_fields, r)
        for r in range(len(optional_fields) + 1)
    ):

        # Generate product of possible values for the current subset of fields
        for values in itertools.product(
            *(get_possible_values(params, field) for field in field_subset)
        ):

            # Create a new permutation by combining base permutation and current field values
            new_perm = {**base_perm, **dict(zip(field_subset, values))}

            yield new_perm


key: _node_content
string_value:
{"id_": "24458b10-beb5-408f-8f7f-68754050e6f2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c6e00e1b-cc88-4465-84ec-6b2a55ffd5b4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "617b5fa7b8277f349b67bea1be74ab702f4797a3754f94a29347390875fb5dd3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66ad3e92-fd1b-4904-aadd-15fc78066604", "node_type": "1", "metadata": {}, "hash": "9ae34d082576d9658dcf480809ba14cfa083c7531ab1b609a59c8979c2f4d6a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33381, "end_char_idx": 36449, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def generate_required_permutations(
    params: Dict[str, Dict[str, Any]], required_fields: List[str]
) -> List[Dict[str, Any]]:
    """
    Generates permutations for the required fields.

    :param params: Parameter dictionary.
    :param required_fields: List of required fields.
    :return: A list of permutations for required fields.
    """

    # Get all possible values for each required field
    required_values = [get_possible_values(params, field) for field in required_fields]

    # Generate permutations from possible values
    return [
        dict(zip(required_fields, values))
        for values in itertools.product(*required_values)
    ]


def generate_optional_permutations(
    params: Dict[str, Dict[str, Any]], base_perm: Dict[str, Any]
) -> Generator[Dict[str, Any], None, None]:
    """
    Generates permutations for optional fields based on a base permutation.

    :param params: Parameter dictionary.
    :param base_perm: Base permutation dictionary.
    :return: A generator yielding each permutation for optional fields.
    """

    # Determine the fields that are optional by subtracting the base permutation's fields from all properties
    optional_fields = set(params["properties"]) - set(base_perm)

    # Iterate through all combinations of optional fields
    for field_subset in itertools.chain.from_iterable(
        itertools.combinations(optional_fields, r)
        for r in range(len(optional_fields) + 1)
    ):

        # Generate product of possible values for the current subset of fields
        for values in itertools.product(
            *(get_possible_values(params, field) for field in field_subset)
        ):

            # Create a new permutation by combining base permutation and current field values
            new_perm = {**base_perm, **dict(zip(field_subset, values))}

            yield new_perm


def get_possible_values(params: Dict[str, Dict[str, Any]], field: str) -> List[Any]:
    """
    Retrieves possible values for a given field.

    :param params: Parameter dictionary.
    :param field: The field for which to get possible values.
    :return: A list of possible values.
    """

    # Extract field information from the parameters
    field_info = params["properties"][field]

    # Based on the field's type or presence of 'enum', determine and return the possible values
    if "enum" in field_info:
        return field_info["enum"]
    elif field_info["type"] == "integer":
        return [placeholder_int]
    elif field_info["type"] == "string":
        return [placeholder_string]
    elif field_info["type"] == "boolean":
        return [True, False]
    elif field_info["type"] == "array" and "enum" in field_info["items"]:
        enum_values = field_info["items"]["enum"]
        all_combinations = [
            list(combo)
            for i in range(1, len(enum_values) + 1)
            for combo in itertools.combinations(enum_values, i)
        ]
        return all_combinations
    return []
```

### Let's generate every invocation for every function first


Prompts:


key: _node_content
string_value:
{"id_": "66ad3e92-fd1b-4904-aadd-15fc78066604", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24458b10-beb5-408f-8f7f-68754050e6f2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2709af6610f4327ba87945793cbdf60bb0764169e9371d8f36d67516c61b3f82", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f8a1de4-eea9-4ba5-a5bd-2af9d207f186", "node_type": "1", "metadata": {}, "hash": "e405244e7d1b5bf45c3de5391ab1c59e1542ccf86725cedafd2bf158f8583767", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35249, "end_char_idx": 37219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def get_possible_values(params: Dict[str, Dict[str, Any]], field: str) -> List[Any]:
    """
    Retrieves possible values for a given field.

    :param params: Parameter dictionary.
    :param field: The field for which to get possible values.
    :return: A list of possible values.
    """

    # Extract field information from the parameters
    field_info = params["properties"][field]

    # Based on the field's type or presence of 'enum', determine and return the possible values
    if "enum" in field_info:
        return field_info["enum"]
    elif field_info["type"] == "integer":
        return [placeholder_int]
    elif field_info["type"] == "string":
        return [placeholder_string]
    elif field_info["type"] == "boolean":
        return [True, False]
    elif field_info["type"] == "array" and "enum" in field_info["items"]:
        enum_values = field_info["items"]["enum"]
        all_combinations = [
            list(combo)
            for i in range(1, len(enum_values) + 1)
            for combo in itertools.combinations(enum_values, i)
        ]
        return all_combinations
    return []
```

### Let's generate every invocation for every function first


Prompts:



```python
INVOCATION_FILLER_PROMPT = """
1) Input reasonable values for 'fill_in_string' and 'fill_in_int' in the invocation here: {invocation}. Reasonable values are determined by the function definition. Use the
the entire function provided here :{function} to get context over what proper fill_in_string and fill_in_int values would be.
Example:

Input: invocation: {{
    "name": "control_camera",
    "arguments": {{
      "mode":"video",
      "duration":"fill_in_int"
    }}
}},
function:{function}

Output: invocation: {{
    "name": "control_camera",
    "arguments": {{
      "mode":"video",
      "duration": 30
    }}
}}


MAKE SURE output is just a dictionary with keys 'name' and 'arguments', no other text or response.

Input: {invocation}
Output:
"""


key: _node_content
string_value:
{"id_": "4f8a1de4-eea9-4ba5-a5bd-2af9d207f186", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66ad3e92-fd1b-4904-aadd-15fc78066604", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f70326b6950d8271ee5ba14609142f99c0c9694a7ab4b2f95a89b96fd1e9b303", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65855be7-e4a1-4050-bddf-66cb8f38b17b", "node_type": "1", "metadata": {}, "hash": "27808f73badd61967c9fdf7f853e4e71bf6186e93700ed7e139d3891994dd25b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36441, "end_char_idx": 39018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Prompts:



```python
INVOCATION_FILLER_PROMPT = """
1) Input reasonable values for 'fill_in_string' and 'fill_in_int' in the invocation here: {invocation}. Reasonable values are determined by the function definition. Use the
the entire function provided here :{function} to get context over what proper fill_in_string and fill_in_int values would be.
Example:

Input: invocation: {{
    "name": "control_camera",
    "arguments": {{
      "mode":"video",
      "duration":"fill_in_int"
    }}
}},
function:{function}

Output: invocation: {{
    "name": "control_camera",
    "arguments": {{
      "mode":"video",
      "duration": 30
    }}
}}


MAKE SURE output is just a dictionary with keys 'name' and 'arguments', no other text or response.

Input: {invocation}
Output:
"""


COMMAND_GENERATION_PROMPT = """
You are to output 2 commands, questions or statements that would generate the inputted function and parameters.
Please make the commands or questions natural, as a person would ask, and the command or questions should be varied and not repetitive.
It should not always mirror the exact technical terminology used in the function and parameters, rather reflect a conversational and intuitive request.
For instance, the prompt should not be 'turn on the dome light', as that is too technical, but rather 'turn on the inside lights'.
Another example, is the prompt should not be 'turn on the HVAC', but rather 'turn on the air conditioning'. Use language a normal driver would use, even if
it is technically incorrect but colloquially used.

RULES: ALWAYS put a backwards slash before an apostrophe or single quote '. For example, do not say don't but say don\'t.
Prompts MUST be in double quotes as well.

Example

Input: {{'name': 'calibrate_sensors','arguments': {{}}'' }}
Prompt: ["The sensors are out of whack, can you reset them", "The calibration of the drone is off, fix it please!"]

Input: {{'name': 'set_autopilot','arguments': {{'status': 'off'}}}}
Prompt: ["OK, I want to take back pilot control now","Turn off the automatic pilot I'm ready control it"]

Input: {invocation}
Prompt:
"""
```

In the below snippet, we generate the invocation of each function except for the `reject_request` function.

To perform effective fine-tuning we need correctly labeled data. We could manually come up with examples and label the data,\
or we can generate synthetic data with the help of `gpt-4o` <br>

Empirically, `gpt-4o` needs a bit more help to get good realistic examples of prompts that would generate the `reject_request` function, so we'll do that next...


key: _node_content
string_value:
{"id_": "65855be7-e4a1-4050-bddf-66cb8f38b17b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f8a1de4-eea9-4ba5-a5bd-2af9d207f186", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b041289c576ac7ca22abcebb5f3e144ad444dc42a562a3ada570c78ae66d68e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c49aea9-5a0b-4bd4-9ec6-1b932c21032e", "node_type": "1", "metadata": {}, "hash": "c91e70ffb9d8ebcf471eeede5800b903e6d6b946e0ecfdb58cfd9d92c9a1948e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39022, "end_char_idx": 42152, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
input_objects = []
all_but_reject = [f for f in function_list if f.get("name") != "reject_request"]

for function in all_but_reject:
    func_name = function["function"]["name"]
    params = function["function"]["parameters"]
    for arguments in generate_permutations(params):
        if any(val in arguments.values() for val in ["fill_in_int", "fill_in_str"]):
            input_object = {"name": func_name, "arguments": arguments}
            messages = [
                {
                    "role": "user",
                    "content": INVOCATION_FILLER_PROMPT.format(
                        invocation=str(input_object), function=function
                    ),
                }
            ]
            input_object, usage = get_chat_completion(
                model="gpt-4o", messages=messages, max_tokens=200, temperature=0.1
            ).content
        else:
            input_object = {"name": func_name, "arguments": arguments}

        input_objects.append(input_object)
```

Now that we have all the invocations, let's use `gpt-4o` to generate prompts that would result in those invocations



```python
def remove_sequences(input_string):
    # Replace the specific sequences with an empty string
    cleaned_string = input_string.replace("```json", "")  # Remove "```json" first
    cleaned_string = cleaned_string.replace("```", "")  # Then remove "```"
    return json.loads(cleaned_string)
```


```python
def create_commands(invocation_list):
    example_list = []
    for i, invocation in enumerate(invocation_list):
        if i < 100:
            print(
                f"\033[34m{np.round(100*i/len(invocation_list),1)}% complete\033[0m")
            if type(invocation) == str or "json" in invocation:
                invocation = remove_sequences(invocation)
            print(invocation)

        # Format the prompt with the invocation string
        request_prompt = COMMAND_GENERATION_PROMPT.format(
            invocation=invocation)

        messages = [{"role": "user", "content": f"{request_prompt}"}]
        completion, usage = get_chat_completion(messages, temperature=0.8)
        command_dict = {"Input": invocation, "Prompt": completion.content}
        example_list.append(command_dict)
    return example_list
```


```python
# Only printing the first 10 rows
training_examples_unformatted = create_commands(input_objects)
```

    [34m0.0% complete[0m
    {'name': 'takeoff_drone', 'arguments': {'altitude': 100}}
    [34m1.8% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'current'}}
    [34m3.5% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'home_base'}}
    [34m5.3% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'custom'}}
    [34m7.0% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'forward', 'distance': 100}}
    [34m8.8% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'backward', 'distance': 50}}
    [34m10.5% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'left', 'distance': 10}}
    [34m12.


key: _node_content
string_value:
{"id_": "4c49aea9-5a0b-4bd4-9ec6-1b932c21032e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65855be7-e4a1-4050-bddf-66cb8f38b17b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "eddc6a883b86e0624cb38eb08bdad1b9e4169b4079e4b29a8e7b4327a272cc44", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08ab6804-9cee-4f46-ab33-7dec60011974", "node_type": "1", "metadata": {}, "hash": "43d820f3573b22bb1ff0913f29f2469154782b2720148138ce6014135eb6a2fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41299, "end_char_idx": 43528, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Only printing the first 10 rows
training_examples_unformatted = create_commands(input_objects)
```

    [34m0.0% complete[0m
    {'name': 'takeoff_drone', 'arguments': {'altitude': 100}}
    [34m1.8% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'current'}}
    [34m3.5% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'home_base'}}
    [34m5.3% complete[0m
    {'name': 'land_drone', 'arguments': {'location': 'custom'}}
    [34m7.0% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'forward', 'distance': 100}}
    [34m8.8% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'backward', 'distance': 50}}
    [34m10.5% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'left', 'distance': 10}}
    [34m12.3% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'right', 'distance': 10}}
    [34m14.0% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'up', 'distance': 10}}
    [34m15.8% complete[0m
    {'name': 'control_drone_movement', 'arguments': {'direction': 'down', 'distance': 10}}
    [34m17.5% complete[0m
    {'name': 'set_drone_speed', 'arguments': {'speed': 10}}
    [34m19.3% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'photo'}}
    [34m21.1% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'photo', 'duration': 10}}
    [34m22.8% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'video'}}
    [34m24.6% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'video', 'duration': 60}}
    [34m26.3% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'panorama'}}
    [34m28.1% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'panorama', 'duration': 60}}
    [34m29.8% complete[0m
    {'name': 'control_gimbal', 'arguments': {'tilt': 45, 'pan': 90}}
    [34m31.6% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'on'}}
    [34m33.3% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'off'}}
    [34m35.1% complete[0m
    {'name': 'set_drone_lighting',


key: _node_content
string_value:
{"id_": "08ab6804-9cee-4f46-ab33-7dec60011974", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c49aea9-5a0b-4bd4-9ec6-1b932c21032e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "fb4080a01123b88c8f1ea82f0723a8bf0c3e7ca206310ed09e46dbebb14ef019", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "743f6d64-5050-4806-9654-25aa35d0b773", "node_type": "1", "metadata": {}, "hash": "008e2e7517cc10270fed48e49c79476a09012fc2428b66838e9d5c8ded11f1c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42444, "end_char_idx": 44579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'arguments': {'direction': 'down', 'distance': 10}}
    [34m17.5% complete[0m
    {'name': 'set_drone_speed', 'arguments': {'speed': 10}}
    [34m19.3% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'photo'}}
    [34m21.1% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'photo', 'duration': 10}}
    [34m22.8% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'video'}}
    [34m24.6% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'video', 'duration': 60}}
    [34m26.3% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'panorama'}}
    [34m28.1% complete[0m
    {'name': 'control_camera', 'arguments': {'mode': 'panorama', 'duration': 60}}
    [34m29.8% complete[0m
    {'name': 'control_gimbal', 'arguments': {'tilt': 45, 'pan': 90}}
    [34m31.6% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'on'}}
    [34m33.3% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'off'}}
    [34m35.1% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'blink'}}
    [34m36.8% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'sos'}}
    [34m38.6% complete[0m
    {'name': 'return_to_home', 'arguments': {}}
    [34m40.4% complete[0m
    {'name': 'set_battery_saver_mode', 'arguments': {'status': 'on'}}
    [34m42.1% complete[0m
    {'name': 'set_battery_saver_mode', 'arguments': {'status': 'off'}}
    [34m43.9% complete[0m
    {'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'on'}}
    [34m45.6% complete[0m
    {'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'off'}}
    [34m47.4% complete[0m
    {'name': 'set_follow_me_mode', 'arguments': {'status': 'on'}}
    [34m49.1% complete[0m
    {'name': 'set_follow_me_mode', 'arguments': {'status': 'off'}}
    [34m50.9% complete[0m
    {'name': 'calibrate_sensors', 'arguments': {}}
    [34m52.6% complete[0m
    {'name': 'set_autopilot', 'arguments': {'status': 'on'}}
    [34m54.4% complete[0m
    {'name': 'set_autopilot', 'arguments': {'status': 'off'}}
    [34m56.


key: _node_content
string_value:
{"id_": "743f6d64-5050-4806-9654-25aa35d0b773", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08ab6804-9cee-4f46-ab33-7dec60011974", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7330aaebab4cf88796aac7eb0e1f6b4ff81b5b69a76130ba934c2651feb82fa1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ab1a439-da04-402c-8243-e0dab12c3c4c", "node_type": "1", "metadata": {}, "hash": "a9b039c718520255cf42e17c97fd0db5a63ff2066ec9a34fecdb98cebfccca4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43478, "end_char_idx": 45728, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'blink'}}
    [34m36.8% complete[0m
    {'name': 'set_drone_lighting', 'arguments': {'mode': 'sos'}}
    [34m38.6% complete[0m
    {'name': 'return_to_home', 'arguments': {}}
    [34m40.4% complete[0m
    {'name': 'set_battery_saver_mode', 'arguments': {'status': 'on'}}
    [34m42.1% complete[0m
    {'name': 'set_battery_saver_mode', 'arguments': {'status': 'off'}}
    [34m43.9% complete[0m
    {'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'on'}}
    [34m45.6% complete[0m
    {'name': 'set_obstacle_avoidance', 'arguments': {'mode': 'off'}}
    [34m47.4% complete[0m
    {'name': 'set_follow_me_mode', 'arguments': {'status': 'on'}}
    [34m49.1% complete[0m
    {'name': 'set_follow_me_mode', 'arguments': {'status': 'off'}}
    [34m50.9% complete[0m
    {'name': 'calibrate_sensors', 'arguments': {}}
    [34m52.6% complete[0m
    {'name': 'set_autopilot', 'arguments': {'status': 'on'}}
    [34m54.4% complete[0m
    {'name': 'set_autopilot', 'arguments': {'status': 'off'}}
    [34m56.1% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid'}}
    [34m57.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'red'}}
    [34m59.6% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'blue'}}
    [34m61.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'green'}}
    [34m63.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'yellow'}}
    [34m64.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'white'}}
    [34m66.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink'}}
    [34m68.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'red'}}
    [34m70.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'blue'}}
    [34m71.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'green'}}
    [34m73.


key: _node_content
string_value:
{"id_": "3ab1a439-da04-402c-8243-e0dab12c3c4c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "743f6d64-5050-4806-9654-25aa35d0b773", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1fa62a088fe696d7e296d2d19e90a299f1348e10b4d037735529fd455ff75dee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a0a45d3-2f45-4ca6-9de0-cacfcdb55962", "node_type": "1", "metadata": {}, "hash": "adbecc63de3ee9606026b33cbd9ab150059511c5047693c9cfefdb1b971dde8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 44535, "end_char_idx": 46936, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'arguments': {'status': 'off'}}
    [34m56.1% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid'}}
    [34m57.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'red'}}
    [34m59.6% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'blue'}}
    [34m61.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'green'}}
    [34m63.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'yellow'}}
    [34m64.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'solid', 'color': 'white'}}
    [34m66.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink'}}
    [34m68.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'red'}}
    [34m70.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'blue'}}
    [34m71.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'green'}}
    [34m73.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'yellow'}}
    [34m75.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'white'}}
    [34m77.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse'}}
    [34m78.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'red'}}
    [34m80.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'blue'}}
    [34m82.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'green'}}
    [34m84.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'yellow'}}
    [34m86.0% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'white'}}
    [34m87.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow'}}
    [34m89.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'red'}}
    [34m91.2% complete[0m
    {'name': 'configure_led_display',


key: _node_content
string_value:
{"id_": "2a0a45d3-2f45-4ca6-9de0-cacfcdb55962", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ab1a439-da04-402c-8243-e0dab12c3c4c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "00f9c6224d7846a9bf4277b5292d845d3e8bf209cfcede926f71ed38971a66bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "588e3d5b-9e0a-42f6-ae15-dd732059febf", "node_type": "1", "metadata": {}, "hash": "8816ecb32fa8870585f9ae01f9369a628ae8caf4197d7a08a01f86b9eb4a220e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45697, "end_char_idx": 48166, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'color': 'green'}}
    [34m73.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'yellow'}}
    [34m75.4% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'blink', 'color': 'white'}}
    [34m77.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse'}}
    [34m78.9% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'red'}}
    [34m80.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'blue'}}
    [34m82.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'green'}}
    [34m84.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'yellow'}}
    [34m86.0% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'pulse', 'color': 'white'}}
    [34m87.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow'}}
    [34m89.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'red'}}
    [34m91.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'blue'}}
    [34m93.0% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'green'}}
    [34m94.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'yellow'}}
    [34m96.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'white'}}
    [34m98.2% complete[0m
    {'name': 'reject_request', 'arguments': {}}


Now let's format the training examples properly. For more documentation on the proper training data formatting for fine tuning for function calling, see here: https://platform.openai.com/docs/guides/fine-tuning/fine-tuning-examples



```python
def remove_descriptions(function_list):
    for function in function_list:
        func = function["function"]
        if "description" in func:
            del func["description"]

        params = func["parameters"]
        if "properties" in params:
            for param in params["properties"].values():
                if "description" in param:
                    del param["description"]

    return function_list


modified_function_list = remove_descriptions(function_list)
```


key: _node_content
string_value:
{"id_": "588e3d5b-9e0a-42f6-ae15-dd732059febf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a0a45d3-2f45-4ca6-9de0-cacfcdb55962", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "51bd54b78d4a131982ac697f265bd236305bb88e5b91de7a66f56344fa9b329b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90e02018-b933-4490-bf7b-245972cd193b", "node_type": "1", "metadata": {}, "hash": "e716f43a2271a9d0f3910f33377ed82f424b99ca1657d32865f5fd208b79489f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 46661, "end_char_idx": 49501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow'}}
    [34m89.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'red'}}
    [34m91.2% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'blue'}}
    [34m93.0% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'green'}}
    [34m94.7% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'yellow'}}
    [34m96.5% complete[0m
    {'name': 'configure_led_display', 'arguments': {'pattern': 'rainbow', 'color': 'white'}}
    [34m98.2% complete[0m
    {'name': 'reject_request', 'arguments': {}}


Now let's format the training examples properly. For more documentation on the proper training data formatting for fine tuning for function calling, see here: https://platform.openai.com/docs/guides/fine-tuning/fine-tuning-examples



```python
def remove_descriptions(function_list):
    for function in function_list:
        func = function["function"]
        if "description" in func:
            del func["description"]

        params = func["parameters"]
        if "properties" in params:
            for param in params["properties"].values():
                if "description" in param:
                    del param["description"]

    return function_list


modified_function_list = remove_descriptions(function_list)
```


```python
training_examples = []

for prompt in training_examples_unformatted:
    # adjust formatting for training data specs

    # if its not a dict, convert to dict
    if type(prompt["Input"]) != dict:
        prompt["Input"] = ast.literal_eval(prompt["Input"])
    prompt["Input"]["arguments"] = json.dumps(prompt["Input"]["arguments"])
    try:
        prompt["Prompt"] = json.loads(prompt["Prompt"])
    except:
        continue
    for p in prompt["Prompt"]:
        print(p)
        print(prompt["Input"])
        tool_calls = [
            {"id": "call_id", "type": "function", "function": prompt["Input"]}
        ]
        training_examples.append(
            {
                "messages": [
                    {"role": "system", "content": DRONE_SYSTEM_PROMPT},
                    {"role": "user", "content": p},
                    {"role": "assistant", "tool_calls": tool_calls},
                ],
                "parallel_tool_calls": False,
                "tools": modified_function_list,
            }
        )
```

    Let's get the drone in the air, how high should it go?
    {'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
    Ready for takeoff, how high should the drone fly?
    {'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
    Can you bring the drone down to where we are?


key: _node_content
string_value:
{"id_": "90e02018-b933-4490-bf7b-245972cd193b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "588e3d5b-9e0a-42f6-ae15-dd732059febf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "501ac0af5cf3fca21abb78d37240ca73b7fa554a3c4bc9c24916ffa9d0ac1517", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "975d703b-6d2d-4f4b-a278-96fe97ac39b0", "node_type": "1", "metadata": {}, "hash": "70f83438c601fcca0b07cf5e787384925b7d813bbf82b7817d7bee5922d85772", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 47668, "end_char_idx": 50870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def remove_descriptions(function_list):
    for function in function_list:
        func = function["function"]
        if "description" in func:
            del func["description"]

        params = func["parameters"]
        if "properties" in params:
            for param in params["properties"].values():
                if "description" in param:
                    del param["description"]

    return function_list


modified_function_list = remove_descriptions(function_list)
```


```python
training_examples = []

for prompt in training_examples_unformatted:
    # adjust formatting for training data specs

    # if its not a dict, convert to dict
    if type(prompt["Input"]) != dict:
        prompt["Input"] = ast.literal_eval(prompt["Input"])
    prompt["Input"]["arguments"] = json.dumps(prompt["Input"]["arguments"])
    try:
        prompt["Prompt"] = json.loads(prompt["Prompt"])
    except:
        continue
    for p in prompt["Prompt"]:
        print(p)
        print(prompt["Input"])
        tool_calls = [
            {"id": "call_id", "type": "function", "function": prompt["Input"]}
        ]
        training_examples.append(
            {
                "messages": [
                    {"role": "system", "content": DRONE_SYSTEM_PROMPT},
                    {"role": "user", "content": p},
                    {"role": "assistant", "tool_calls": tool_calls},
                ],
                "parallel_tool_calls": False,
                "tools": modified_function_list,
            }
        )
```

    Let's get the drone in the air, how high should it go?
    {'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
    Ready for takeoff, how high should the drone fly?
    {'name': 'takeoff_drone', 'arguments': '{"altitude": 100}'}
    Can you bring the drone down to where we are?
    {'name': 'land_drone', 'arguments': '{"location": "current"}'}
    Let's get the drone to land right here
    {'name': 'land_drone', 'arguments': '{"location": "current"}'}
    Bring the drone back to base for landing
    {'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
    Can you safely land the drone at home base
    {'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
    Can you make the drone move to the left by 10 units?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
    I need the drone to go left, could you move it 10 steps that way?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
    Can you move the drone to the right by 10 feet?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
    I need the drone to go 10 feet to the right, can you do that?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
    Can you make the drone go upwards by 10 units?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
    I need the drone to move up, can you do that for me?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
    Can you bring the drone lower by 10 feet please?


key: _node_content
string_value:
{"id_": "975d703b-6d2d-4f4b-a278-96fe97ac39b0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90e02018-b933-4490-bf7b-245972cd193b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3a20c154e8ad57e811641d5612ed46760fd3d8d6574a63703f01ea9b602d2504", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7648fdfd-33c0-40d8-9c99-b62e5bdc54b6", "node_type": "1", "metadata": {}, "hash": "a5cd2e96efeddbd49fc117a07af801f025551ecc7f1a3bfe4af2f05763d6439a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49506, "end_char_idx": 52116, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'land_drone', 'arguments': '{"location": "current"}'}
    Let's get the drone to land right here
    {'name': 'land_drone', 'arguments': '{"location": "current"}'}
    Bring the drone back to base for landing
    {'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
    Can you safely land the drone at home base
    {'name': 'land_drone', 'arguments': '{"location": "home_base"}'}
    Can you make the drone move to the left by 10 units?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
    I need the drone to go left, could you move it 10 steps that way?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "left", "distance": 10}'}
    Can you move the drone to the right by 10 feet?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
    I need the drone to go 10 feet to the right, can you do that?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "right", "distance": 10}'}
    Can you make the drone go upwards by 10 units?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
    I need the drone to move up, can you do that for me?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
    Can you bring the drone lower by 10 feet please?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
    I need the drone to descend 10 units, can you make that happen?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
    Can you make the drone go faster?
    {'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
    I think the drone should speed up a bit, don't you think?
    {'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
    I want to take a picture, can you switch the camera mode to photo
    {'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
    Let's capture this moment, switch the camera to photo mode please
    {'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
    Can you switch the camera to photo mode and take a picture for 10 seconds?
    {'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
    I need to capture something, can you set the camera to take photos for 10 seconds?
    {'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
    Can you switch the camera to video mode?
    {'name': 'control_camera', 'arguments': '{"mode": "video"}'}
    I want to record, can you set the camera to video mode?


key: _node_content
string_value:
{"id_": "7648fdfd-33c0-40d8-9c99-b62e5bdc54b6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "975d703b-6d2d-4f4b-a278-96fe97ac39b0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b8057e4b7be08dd4d3a10fda825995c8057e5a7b985ad09914254d2603703a6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb3976f3-795f-4430-a8c7-4efa73b9afd9", "node_type": "1", "metadata": {}, "hash": "52b3f8c9622b91912f96a0a2f130f6b604a957026432ba6e0a556a07d9f4e2e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 50731, "end_char_idx": 53272, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'control_drone_movement', 'arguments': '{"direction": "up", "distance": 10}'}
    Can you bring the drone lower by 10 feet please?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
    I need the drone to descend 10 units, can you make that happen?
    {'name': 'control_drone_movement', 'arguments': '{"direction": "down", "distance": 10}'}
    Can you make the drone go faster?
    {'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
    I think the drone should speed up a bit, don't you think?
    {'name': 'set_drone_speed', 'arguments': '{"speed": 10}'}
    I want to take a picture, can you switch the camera mode to photo
    {'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
    Let's capture this moment, switch the camera to photo mode please
    {'name': 'control_camera', 'arguments': '{"mode": "photo"}'}
    Can you switch the camera to photo mode and take a picture for 10 seconds?
    {'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
    I need to capture something, can you set the camera to take photos for 10 seconds?
    {'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
    Can you switch the camera to video mode?
    {'name': 'control_camera', 'arguments': '{"mode": "video"}'}
    I want to record, can you set the camera to video mode?
    {'name': 'control_camera', 'arguments': '{"mode": "video"}'}
    Can you start recording a video with the camera for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
    I need to film something, can you put the camera in video mode for 60 seconds
    {'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
    Can you switch the camera to panorama mode?
    {'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
    I'd like to take a 360-degree photo, can you set the camera to panorama mode?
    {'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
    Can you set the camera to take a panorama shot for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
    I'd like to switch the camera mode to panorama and have it last for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
    Can you adjust the camera angle up and to the right?
    {'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
    I need to tilt the camera up and pan it to the right, can you do that?


key: _node_content
string_value:
{"id_": "bb3976f3-795f-4430-a8c7-4efa73b9afd9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7648fdfd-33c0-40d8-9c99-b62e5bdc54b6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8c41185656516c8572c43273bd9ba58c5a6fc69a38b8d9cfe00e20246fc8f605", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e648d2f-3a85-430c-82ad-5e9d1a7ded16", "node_type": "1", "metadata": {}, "hash": "d76124c319154f2dbca48554e8f2fc199a43c604583ffdd7fa816c9f00114440", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51870, "end_char_idx": 54462, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'control_camera', 'arguments': '{"mode": "photo", "duration": 10}'}
    Can you switch the camera to video mode?
    {'name': 'control_camera', 'arguments': '{"mode": "video"}'}
    I want to record, can you set the camera to video mode?
    {'name': 'control_camera', 'arguments': '{"mode": "video"}'}
    Can you start recording a video with the camera for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
    I need to film something, can you put the camera in video mode for 60 seconds
    {'name': 'control_camera', 'arguments': '{"mode": "video", "duration": 60}'}
    Can you switch the camera to panorama mode?
    {'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
    I'd like to take a 360-degree photo, can you set the camera to panorama mode?
    {'name': 'control_camera', 'arguments': '{"mode": "panorama"}'}
    Can you set the camera to take a panorama shot for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
    I'd like to switch the camera mode to panorama and have it last for a minute
    {'name': 'control_camera', 'arguments': '{"mode": "panorama", "duration": 60}'}
    Can you adjust the camera angle up and to the right?
    {'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
    I need to tilt the camera up and pan it to the right, can you do that?
    {'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
    Can you turn on the lights for the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
    I need some extra light, can you activate it on the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
    Can you turn off the lights on the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
    I don't need the drone lights on, can you switch them off
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
    Can you make the drone lights flash?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
    I want the drone lights to blink, can you do that?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
    Can you switch the drone lights to the SOS mode, just in case?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
    I need the drone lights to flash SOS, can you set that up?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
    Can you bring the drone back home now?
    {'name': 'return_to_home', 'arguments': '{}'}
    Is it time for the drone to return to base?


key: _node_content
string_value:
{"id_": "7e648d2f-3a85-430c-82ad-5e9d1a7ded16", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb3976f3-795f-4430-a8c7-4efa73b9afd9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e98b60f7f279f9020fd39670b53cf9eb83e7406177820b0c481d5bd0c79fb61b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7bff103d-7df0-4355-806f-9d92315ce88f", "node_type": "1", "metadata": {}, "hash": "1a6bec9c48cf99d74fd61d434e81a60aaa3ddd01d9c3810b254e5ab9cbd55a46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53131, "end_char_idx": 55842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
    I need to tilt the camera up and pan it to the right, can you do that?
    {'name': 'control_gimbal', 'arguments': '{"tilt": 45, "pan": 90}'}
    Can you turn on the lights for the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
    I need some extra light, can you activate it on the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "on"}'}
    Can you turn off the lights on the drone
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
    I don't need the drone lights on, can you switch them off
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "off"}'}
    Can you make the drone lights flash?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
    I want the drone lights to blink, can you do that?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "blink"}'}
    Can you switch the drone lights to the SOS mode, just in case?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
    I need the drone lights to flash SOS, can you set that up?
    {'name': 'set_drone_lighting', 'arguments': '{"mode": "sos"}'}
    Can you bring the drone back home now?
    {'name': 'return_to_home', 'arguments': '{}'}
    Is it time for the drone to return to base?
    {'name': 'return_to_home', 'arguments': '{}'}
    My phone battery is draining so fast, can you turn on battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
    I need my laptop battery to last longer, can you switch on battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
    My phone battery is draining too quickly, can you turn off the battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
    I feel like my device is slower with battery saver on, can we turn it off?
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
    I want the car to avoid obstacles, can you turn on that feature?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
    Can you activate the obstacle avoidance mode for safety purposes?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
    I'd like to turn off obstacle detection, how do I do that?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
    Can you disable the obstacle avoidance feature for now?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
    Can you activate the follow me mode?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
    I want the car to follow me, can you turn on that feature?


key: _node_content
string_value:
{"id_": "7bff103d-7df0-4355-806f-9d92315ce88f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e648d2f-3a85-430c-82ad-5e9d1a7ded16", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "80462b3ea1f80f69830fbd7333a1a3c9ef6ea5be5cf4fdb6d47ed859a17a7056", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e998f655-948f-4a3a-aa5d-06baadc4d875", "node_type": "1", "metadata": {}, "hash": "797aab6d7d11d1d1075087796c844f8ae191289886b6f95d598b867a854e7d66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 54369, "end_char_idx": 57087, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'return_to_home', 'arguments': '{}'}
    Is it time for the drone to return to base?
    {'name': 'return_to_home', 'arguments': '{}'}
    My phone battery is draining so fast, can you turn on battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
    I need my laptop battery to last longer, can you switch on battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "on"}'}
    My phone battery is draining too quickly, can you turn off the battery saver mode
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
    I feel like my device is slower with battery saver on, can we turn it off?
    {'name': 'set_battery_saver_mode', 'arguments': '{"status": "off"}'}
    I want the car to avoid obstacles, can you turn on that feature?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
    Can you activate the obstacle avoidance mode for safety purposes?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "on"}'}
    I'd like to turn off obstacle detection, how do I do that?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
    Can you disable the obstacle avoidance feature for now?
    {'name': 'set_obstacle_avoidance', 'arguments': '{"mode": "off"}'}
    Can you activate the follow me mode?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
    I want the car to follow me, can you turn on that feature?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
    I don't want the drone following me anymore, can you turn that off?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
    Can you disable the follow-me mode on the drone?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
    The sensors are acting up, can you recalibrate them
    {'name': 'calibrate_sensors', 'arguments': '{}'}
    My device doesn't seem to be sensing correctly, can you adjust it
    {'name': 'calibrate_sensors', 'arguments': '{}'}
    I'm too tired to drive, can you turn on the autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
    Let the car drive itself, turn on autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
    I'm feeling more confident, turn off the autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
    I think I can handle it, deactivate the automatic pilot
    {'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
    Can you set the display to a steady yellow color?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
    I'd like the LED display to be a solid yellow, please.


key: _node_content
string_value:
{"id_": "e998f655-948f-4a3a-aa5d-06baadc4d875", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7bff103d-7df0-4355-806f-9d92315ce88f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "21567382f756c93dbda5bdec1d279ba718c7187198a5f96fb95d06648388ed38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "759f3132-e099-44c5-a1bc-4cb0d5ed692e", "node_type": "1", "metadata": {}, "hash": "0e3df9ed3688dbde6e4c977b6bf84579733b0b3c71682a44a843005124c17d1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55716, "end_char_idx": 58453, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
    I want the car to follow me, can you turn on that feature?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "on"}'}
    I don't want the drone following me anymore, can you turn that off?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
    Can you disable the follow-me mode on the drone?
    {'name': 'set_follow_me_mode', 'arguments': '{"status": "off"}'}
    The sensors are acting up, can you recalibrate them
    {'name': 'calibrate_sensors', 'arguments': '{}'}
    My device doesn't seem to be sensing correctly, can you adjust it
    {'name': 'calibrate_sensors', 'arguments': '{}'}
    I'm too tired to drive, can you turn on the autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
    Let the car drive itself, turn on autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "on"}'}
    I'm feeling more confident, turn off the autopilot
    {'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
    I think I can handle it, deactivate the automatic pilot
    {'name': 'set_autopilot', 'arguments': '{"status": "off"}'}
    Can you set the display to a steady yellow color?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
    I'd like the LED display to be a solid yellow, please.
    {'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
    Can you make the lights flash on and off
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
    I want the LED display to blink, can you set that up
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
    Can you make the lights flash in red?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
    How do I set the display to blink in red?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
    Can you make the lights flash in yellow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
    How do I set the display to blink in yellow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
    Can you make the lights blink instead of staying steady
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
    I want the LEDs to flash, not stay solid
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
    Can you make the LED display pulse in red, please?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
    I'd like the LED display to flash in red, can you set that up?


key: _node_content
string_value:
{"id_": "759f3132-e099-44c5-a1bc-4cb0d5ed692e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e998f655-948f-4a3a-aa5d-06baadc4d875", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "062795fd9f6d5890342e622787bbf73e6bee9606d5007ee77cbfe8fe1a222766", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f04db181-3eca-4692-a79c-2f7738219ea0", "node_type": "1", "metadata": {}, "hash": "37948defa57c3dc413b978b059d8fbc1954583b7a938d988b9dd7ad53131799c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 56939, "end_char_idx": 59865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
    I'd like the LED display to be a solid yellow, please.
    {'name': 'configure_led_display', 'arguments': '{"pattern": "solid", "color": "yellow"}'}
    Can you make the lights flash on and off
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
    I want the LED display to blink, can you set that up
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink"}'}
    Can you make the lights flash in red?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
    How do I set the display to blink in red?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "red"}'}
    Can you make the lights flash in yellow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
    How do I set the display to blink in yellow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "blink", "color": "yellow"}'}
    Can you make the lights blink instead of staying steady
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
    I want the LEDs to flash, not stay solid
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse"}'}
    Can you make the LED display pulse in red, please?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
    I'd like the LED display to flash in red, can you set that up?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
    I want the LED lights to flash in blue
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
    Can you set the display to pulse with a blue color
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
    Can you make the lights flash and change to green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
    Let's set the LEDs to blink and switch to green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
    Can you change the flashy lights to yellow and make them pulse
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
    I want the LED display to blink in yellow, can you do that
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
    Can you change the colors on the display to red and set it to a rainbow pattern?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
    I want the LED display to show a rainbow pattern in red, can you set that up?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
    Can you change the color and pattern of the lights to blue and rainbow?


key: _node_content
string_value:
{"id_": "f04db181-3eca-4692-a79c-2f7738219ea0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "759f3132-e099-44c5-a1bc-4cb0d5ed692e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1297e2001926f6db9812c77c5bc33a20d353d820b746be3c76ded3ae9075eadb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6b15b0e-12af-45b6-b06a-8b7504f647a3", "node_type": "1", "metadata": {}, "hash": "544c704c8cfb682d49df0b297cafcfdaea993250985d47f999b0d519e39c011a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 58458, "end_char_idx": 61305, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "red"}'}
    I want the LED lights to flash in blue
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
    Can you set the display to pulse with a blue color
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "blue"}'}
    Can you make the lights flash and change to green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
    Let's set the LEDs to blink and switch to green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "green"}'}
    Can you change the flashy lights to yellow and make them pulse
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
    I want the LED display to blink in yellow, can you do that
    {'name': 'configure_led_display', 'arguments': '{"pattern": "pulse", "color": "yellow"}'}
    Can you change the colors on the display to red and set it to a rainbow pattern?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
    I want the LED display to show a rainbow pattern in red, can you set that up?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
    Can you change the color and pattern of the lights to blue and rainbow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
    I'm feeling like some colorful lights, can you set it to blue and rainbow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
    Can you set the LED display to show a rainbow pattern in green color?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
    I'd like the LED display to cycle through colors, starting with green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
    Can you make the lights do a cool rainbow effect
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
    Change the color of the lights to white and make them change like a rainbow
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
    I changed my mind, can you cancel that request
    {'name': 'reject_request', 'arguments': '{}'}
    I don't want to proceed with the request anymore, can you reject it
    {'name': 'reject_request', 'arguments': '{}'}


Now, back to the rejection function. Let's generate some prompts that are _nearly_ possible, but should result in the `reject_request` function being called. To do so, we queried `gpt-4o` asking for requests that are related to, but not quite possible with, the given list of functions.


key: _node_content
string_value:
{"id_": "f6b15b0e-12af-45b6-b06a-8b7504f647a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f04db181-3eca-4692-a79c-2f7738219ea0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2f15b4fa2755b3c7615507f503d185658e87b421786cef4883870a0279f83cc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a30c666-46e0-4783-9245-192971f1cde1", "node_type": "1", "metadata": {}, "hash": "d37fa88715bc67e4b7c7ff0f1153428d21fd61ef017f0faed51973022612a0f0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 59701, "end_char_idx": 62894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "red"}'}
    Can you change the color and pattern of the lights to blue and rainbow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
    I'm feeling like some colorful lights, can you set it to blue and rainbow?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "blue"}'}
    Can you set the LED display to show a rainbow pattern in green color?
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
    I'd like the LED display to cycle through colors, starting with green
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "green"}'}
    Can you make the lights do a cool rainbow effect
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
    Change the color of the lights to white and make them change like a rainbow
    {'name': 'configure_led_display', 'arguments': '{"pattern": "rainbow", "color": "white"}'}
    I changed my mind, can you cancel that request
    {'name': 'reject_request', 'arguments': '{}'}
    I don't want to proceed with the request anymore, can you reject it
    {'name': 'reject_request', 'arguments': '{}'}


Now, back to the rejection function. Let's generate some prompts that are _nearly_ possible, but should result in the `reject_request` function being called. To do so, we queried `gpt-4o` asking for requests that are related to, but not quite possible with, the given list of functions.



```python
reject_list = [
    "Translate broadcast message to another language",
    "Automatically capture photos when face is detected",
    "Detect nearby drones",
    "Measure wind resistance",
    "Capture slow motion video",
    "Move the drone forward and backward by same distance at the same time.",
    "Adjust drone's altitude to ground level changes",
    "Display custom message on LED display",
    "Sync drone's time with smartphone",
    "Alert when drone travels out of designated area",
    "Calibrate sensors and land simultaneously",
    "Detect moisture levels",
    "Automatically follow GPS tagged object",
    "Toggle night vision mode",
    "Maintain current altitude when battery is low",
    "Decide best landing spot using AI",
    "Program drone's route based on wind direction",
]
```


```python
reject_training_list = []
for prompt in reject_list:
    # Adjust formatting
    tool_calls = [
        {
            "id": "call_id",
            "type": "function",
            "function": {"name": "reject_request", "arguments": "{}"},
        }
    ]
    reject_training_list.append(
        {
            "messages": [
                {"role": "system", "content": DRONE_SYSTEM_PROMPT},
                {"role": "user", "content": prompt},
                {"role": "assistant", "tool_calls": tool_calls},
            ],
            "parallel_tool_calls": False,
            "tools": modified_function_list,
        }
    )
```

Now combine all the training examples together



```python
training_list_total = training_examples + reject_training_list
```


key: _node_content
string_value:
{"id_": "5a30c666-46e0-4783-9245-192971f1cde1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6b15b0e-12af-45b6-b06a-8b7504f647a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3105369373818b7df24251ccb34cd061c32fc3fa6720907cb11ace484f2c67a1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d311ca04-9f62-4b79-b3f9-e70efbab007b", "node_type": "1", "metadata": {}, "hash": "8ee4aa8f0781deaadc80c94163088e4a70694cf7d6289efa93755013f883fd6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 61019, "end_char_idx": 63158, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now, back to the rejection function. Let's generate some prompts that are _nearly_ possible, but should result in the `reject_request` function being called. To do so, we queried `gpt-4o` asking for requests that are related to, but not quite possible with, the given list of functions.



```python
reject_list = [
    "Translate broadcast message to another language",
    "Automatically capture photos when face is detected",
    "Detect nearby drones",
    "Measure wind resistance",
    "Capture slow motion video",
    "Move the drone forward and backward by same distance at the same time.",
    "Adjust drone's altitude to ground level changes",
    "Display custom message on LED display",
    "Sync drone's time with smartphone",
    "Alert when drone travels out of designated area",
    "Calibrate sensors and land simultaneously",
    "Detect moisture levels",
    "Automatically follow GPS tagged object",
    "Toggle night vision mode",
    "Maintain current altitude when battery is low",
    "Decide best landing spot using AI",
    "Program drone's route based on wind direction",
]
```


```python
reject_training_list = []
for prompt in reject_list:
    # Adjust formatting
    tool_calls = [
        {
            "id": "call_id",
            "type": "function",
            "function": {"name": "reject_request", "arguments": "{}"},
        }
    ]
    reject_training_list.append(
        {
            "messages": [
                {"role": "system", "content": DRONE_SYSTEM_PROMPT},
                {"role": "user", "content": prompt},
                {"role": "assistant", "tool_calls": tool_calls},
            ],
            "parallel_tool_calls": False,
            "tools": modified_function_list,
        }
    )
```

Now combine all the training examples together



```python
training_list_total = training_examples + reject_training_list
```


```python
training_file = "data/drone_training.jsonl"
with open(training_file, "w") as f:
    for item in training_list_total:
        json_str = json.dumps(item)
        f.write(f"{json_str}\n")
```

# Fine tuning


Finally, we can kick off the fine-tuning job


key: _node_content
string_value:
{"id_": "d311ca04-9f62-4b79-b3f9-e70efbab007b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a30c666-46e0-4783-9245-192971f1cde1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "cf64f60e27f38f2979a353d029153f5c7d994c23549bae796a8bc1a2d97edb5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "654a7a2c-a673-4305-b431-d2b64213b9e6", "node_type": "1", "metadata": {}, "hash": "0e32d6c301c6ed367dd287503dcdf5f16e20361765e7d170c43ff05f9dd8a968", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 62126, "end_char_idx": 64872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
reject_training_list = []
for prompt in reject_list:
    # Adjust formatting
    tool_calls = [
        {
            "id": "call_id",
            "type": "function",
            "function": {"name": "reject_request", "arguments": "{}"},
        }
    ]
    reject_training_list.append(
        {
            "messages": [
                {"role": "system", "content": DRONE_SYSTEM_PROMPT},
                {"role": "user", "content": prompt},
                {"role": "assistant", "tool_calls": tool_calls},
            ],
            "parallel_tool_calls": False,
            "tools": modified_function_list,
        }
    )
```

Now combine all the training examples together



```python
training_list_total = training_examples + reject_training_list
```


```python
training_file = "data/drone_training.jsonl"
with open(training_file, "w") as f:
    for item in training_list_total:
        json_str = json.dumps(item)
        f.write(f"{json_str}\n")
```

# Fine tuning


Finally, we can kick off the fine-tuning job



```python
# Upload the training file
file = client.files.create(
    file=open("data/drone_training.jsonl", "rb"),
    purpose="fine-tune",
)
file_id = file.id
print(f"FileID: {file_id}")

# Create a fine-tuning job

ft = client.fine_tuning.jobs.create(
    model="gpt-3.5-turbo",
    training_file=file_id,
    suffix="drone",
)

print(f"Fine-tuning job created: {ft}")
```

    FileID: file-blg0IytwIivZQzc9mbfnS8Pm
    Fine-tuning job created: FineTuningJob(id='ftjob-84PQg97hoIAKf21IPnhiNlU1', created_at=1718580285, error=Error(code=None, message=None, param=None), fine_tuned_model=None, finished_at=None, hyperparameters=Hyperparameters(n_epochs='auto', batch_size='auto', learning_rate_multiplier='auto'), model='gpt-3.5-turbo-0125', object='fine_tuning.job', organization_id='org-lb41cclBdkq5pm6BgDhx8DHP', result_files=[], seed=1513865891, status='validating_files', trained_tokens=None, training_file='file-blg0IytwIivZQzc9mbfnS8Pm', validation_file=None, estimated_finish=None, integrations=[], user_provided_suffix='drone')


In addition to creating a fine-tuning job, you can also list existing jobs, retrieve the status of a job, or cancel a job.



```python
ftjob_id = "ftjob-84PQg97hoIAKf21IPnhiNlU1"
# List 10 fine-tuning jobs
# client.fine_tuning.jobs.list(limit=10)

# Retrieve the state of a fine-tune
client.fine_tuning.jobs.retrieve(ftjob_id)

# Cancel a job
# client.fine_tuning.jobs.cancel("ftjob-abc123")

# List up to 10 events from a fine-tuning job
# client.fine_tuning.jobs.list_events(fine_tuning_job_id="ftjob-abc123", limit=10)

# Delete a fine-tuned model (must be an owner of the org the model was created in)
# client.models.delete("ft:gpt-3.5-turbo:abc:suffix:abc123")
```


key: _node_content
string_value:
{"id_": "654a7a2c-a673-4305-b431-d2b64213b9e6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d311ca04-9f62-4b79-b3f9-e70efbab007b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "07b412493e505cc2ed60527a4dc44afe9f8f632e86588d9b4baed0ee9a746258", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a30a7344-0b3c-4ea7-828e-7999ea0480ff", "node_type": "1", "metadata": {}, "hash": "ed10c97f446ea5370f11505fba19064dd2f9761643787c41d01f58ea4522fa88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In addition to creating a fine-tuning job, you can also list existing jobs, retrieve the status of a job, or cancel a job.



```python
ftjob_id = "ftjob-84PQg97hoIAKf21IPnhiNlU1"
# List 10 fine-tuning jobs
# client.fine_tuning.jobs.list(limit=10)

# Retrieve the state of a fine-tune
client.fine_tuning.jobs.retrieve(ftjob_id)

# Cancel a job
# client.fine_tuning.jobs.cancel("ftjob-abc123")

# List up to 10 events from a fine-tuning job
# client.fine_tuning.jobs.list_events(fine_tuning_job_id="ftjob-abc123", limit=10)

# Delete a fine-tuned model (must be an owner of the org the model was created in)
# client.models.delete("ft:gpt-3.5-turbo:abc:suffix:abc123")
```




    FineTuningJob(id='ftjob-84PQg97hoIAKf21IPnhiNlU1', created_at=1718580285, error=Error(code=None, message=None, param=None), fine_tuned_model='ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC', finished_at=1718581004, hyperparameters=Hyperparameters(n_epochs=3, batch_size=1, learning_rate_multiplier=2), model='gpt-3.5-turbo-0125', object='fine_tuning.job', organization_id='org-lb41cclBdkq5pm6BgDhx8DHP', result_files=['file-F6XPJFLVG9f3mR04KBmwUI9H'], seed=1513865891, status='succeeded', trained_tokens=145983, training_file='file-blg0IytwIivZQzc9mbfnS8Pm', validation_file=None, estimated_finish=None, integrations=[], user_provided_suffix='drone')



After a fine-tuning job has finished, you can also see metrics around how the training process went by querying a fine-tuning job, extracting a file ID from the result_files, and then retrieving that files content. Each results CSV file has the following columns: step, train_loss, train_accuracy, valid_loss, and valid_mean_token_accuracy. While metrics can he helpful, evaluating samples from the fine-tuned model provides the most relevant sense of model quality.



```python
fine_tune_results = client.fine_tuning.jobs.retrieve(ftjob_id).result_files
result_file_id = client.files.retrieve(fine_tune_results[0]).id

# Retrieve the result file
result_file = client.files.content(file_id=result_file_id)
decoded_content = base64.b64decode(result_file.read()).decode("utf-8")
print(decoded_content)
```

    step,train_loss,train_accuracy,valid_loss,valid_mean_token_accuracy
    1,3.63265,0.5,
    2,2.45992,0.80952,
    3,2.77939,0.80952,
    4,3.53073,0.65,
    5,2.61654,0.8,
    6,2.16,0.85714,
    7,2.73706,0.8,
    8,2.56944,0.625,
    9,2.06096,0.78947,


key: _node_content
string_value:
{"id_": "a30a7344-0b3c-4ea7-828e-7999ea0480ff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "654a7a2c-a673-4305-b431-d2b64213b9e6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0702989d9cf90f50b8800df153193d2ef80fee951b87f6ab3db6f5d76961d4b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12351208-b28e-4ac7-b332-ef0b64533741", "node_type": "1", "metadata": {}, "hash": "6ff5d6ca3070ea80a38552907819697d6cefb2c5ba678771185038dccbfe2f22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
After a fine-tuning job has finished, you can also see metrics around how the training process went by querying a fine-tuning job, extracting a file ID from the result_files, and then retrieving that files content. Each results CSV file has the following columns: step, train_loss, train_accuracy, valid_loss, and valid_mean_token_accuracy. While metrics can he helpful, evaluating samples from the fine-tuned model provides the most relevant sense of model quality.



```python
fine_tune_results = client.fine_tuning.jobs.retrieve(ftjob_id).result_files
result_file_id = client.files.retrieve(fine_tune_results[0]).id

# Retrieve the result file
result_file = client.files.content(file_id=result_file_id)
decoded_content = base64.b64decode(result_file.read()).decode("utf-8")
print(decoded_content)
```

    step,train_loss,train_accuracy,valid_loss,valid_mean_token_accuracy
    1,3.63265,0.5,
    2,2.45992,0.80952,
    3,2.77939,0.80952,
    4,3.53073,0.65,
    5,2.61654,0.8,
    6,2.16,0.85714,
    7,2.73706,0.8,
    8,2.56944,0.625,
    9,2.06096,0.78947,
    10,1.69598,0.8,
    11,1.94268,0.77778,
    12,1.61752,0.86667,
    13,1.2442,0.8,
    14,0.73411,0.875,
    15,0.34285,0.875,
    16,0.22229,0.95238,
    17,0.04635,0.95,
    18,0.00626,1.0,
    19,0.60888,0.90909,
    20,0.00092,1.0,
    21,0.8001,0.95,
    22,0.04982,1.0,
    23,0.35494,0.92857,
    24,0.00023,1.0,
    25,0.00034,1.0,
    26,0.0029,1.0,
    27,0.58017,0.875,
    28,0.13018,0.9375,
    29,0.00109,1.0,
    30,6e-05,1.0,
    31,0.61665,0.95,
    32,3e-05,1.0,
    33,0.23598,0.95,
    34,3e-05,1.0,
    35,0.03566,1.0,
    36,1e-05,1.0,
    37,1e-05,1.0,
    38,2e-05,1.0,
    39,2e-05,1.0,
    40,0.00034,1.0,


key: _node_content
string_value:
{"id_": "12351208-b28e-4ac7-b332-ef0b64533741", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a30a7344-0b3c-4ea7-828e-7999ea0480ff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "67c39b29a6a1abcbdf0ccad449885463b12648b0098cbd290ad575a64c98185e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "142b9225-fb7e-455b-8511-24f53ac59b7e", "node_type": "1", "metadata": {}, "hash": "ba9e5b58a54751dad74772258f171edc70d5ab149e399d7b60c45916cdb1b08b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
86667,
    13,1.2442,0.8,
    14,0.73411,0.875,
    15,0.34285,0.875,
    16,0.22229,0.95238,
    17,0.04635,0.95,
    18,0.00626,1.0,
    19,0.60888,0.90909,
    20,0.00092,1.0,
    21,0.8001,0.95,
    22,0.04982,1.0,
    23,0.35494,0.92857,
    24,0.00023,1.0,
    25,0.00034,1.0,
    26,0.0029,1.0,
    27,0.58017,0.875,
    28,0.13018,0.9375,
    29,0.00109,1.0,
    30,6e-05,1.0,
    31,0.61665,0.95,
    32,3e-05,1.0,
    33,0.23598,0.95,
    34,3e-05,1.0,
    35,0.03566,1.0,
    36,1e-05,1.0,
    37,1e-05,1.0,
    38,2e-05,1.0,
    39,2e-05,1.0,
    40,0.00034,1.0,
    41,0.0,1.0,
    42,0.0,1.0,
    43,0.0,1.0,
    44,0.0,1.0,
    45,0.0,1.0,
    46,0.91896,0.95,
    47,0.0,1.0,
    48,0.12006,0.95,
    49,0.0,1.0,
    50,3.92872,0.75,
    51,0.0,1.0,
    52,0.98277,0.90476,
    53,0.0,1.0,
    54,0.0,1.0,
    55,1e-05,1.0,
    56,0.00401,1.0,
    57,0.07366,1.0,
    58,0.0,1.0,
    59,0.0,1.0,
    60,0.0,1.0,
    61,0.0,1.0,
    62,0.10347,0.875,
    63,0.0,1.0,
    64,0.0,1.0,
    65,1e-05,1.0,
    66,2.97112,0.85714,
    67,1.12396,0.875,
    68,2e-05,


key: _node_content
string_value:
{"id_": "142b9225-fb7e-455b-8511-24f53ac59b7e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12351208-b28e-4ac7-b332-ef0b64533741", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4032a72794f271f77434e5a2493cf41f969c40c36ee9a150c199b2aac47760a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe58b4d0-cd36-45ff-be66-850bf7bb24fb", "node_type": "1", "metadata": {}, "hash": "c0c71ec11b527bc840168a4c135b213a458783e152d5ee6931b64ae4c7d40822", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2e-05,1.0,
    40,0.00034,1.0,
    41,0.0,1.0,
    42,0.0,1.0,
    43,0.0,1.0,
    44,0.0,1.0,
    45,0.0,1.0,
    46,0.91896,0.95,
    47,0.0,1.0,
    48,0.12006,0.95,
    49,0.0,1.0,
    50,3.92872,0.75,
    51,0.0,1.0,
    52,0.98277,0.90476,
    53,0.0,1.0,
    54,0.0,1.0,
    55,1e-05,1.0,
    56,0.00401,1.0,
    57,0.07366,1.0,
    58,0.0,1.0,
    59,0.0,1.0,
    60,0.0,1.0,
    61,0.0,1.0,
    62,0.10347,0.875,
    63,0.0,1.0,
    64,0.0,1.0,
    65,1e-05,1.0,
    66,2.97112,0.85714,
    67,1.12396,0.875,
    68,2e-05,1.0,
    69,0.00067,1.0,
    70,0.0,1.0,
    71,0.0,1.0,
    72,0.0,1.0,
    73,0.0,1.0,
    74,0.0,1.0,
    75,0.02064,1.0,
    76,0.5146,0.86667,
    77,0.18756,0.95,
    78,6e-05,1.0,
    79,0.0,1.0,
    80,0.21298,0.93333,
    81,0.0,1.0,
    82,0.0,1.0,
    83,0.0,1.0,
    84,0.00139,1.0,
    85,0.0,1.0,
    86,0.85297,0.875,
    87,0.0,1.0,
    88,0.0,1.0,
    89,1.45164,0.875,
    90,0.0,1.0,
    91,0.05329,0.92857,
    92,0.55506,0.93333,
    93,0.42187,0.92857,
    94,0.0,1.0,
    95,0.0,1.0,


key: _node_content
string_value:
{"id_": "fe58b4d0-cd36-45ff-be66-850bf7bb24fb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "142b9225-fb7e-455b-8511-24f53ac59b7e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d5f1c134bbf66837895600107d00643e9163fa0f784ef1f537beda53c2425d4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a2ab504-464e-4f9d-95a1-bde4dd212db9", "node_type": "1", "metadata": {}, "hash": "658db7898fc4f64fa195b9bea209c4f54dceef54d0130a0b1b7654fbbe4f7b1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
85714,
    67,1.12396,0.875,
    68,2e-05,1.0,
    69,0.00067,1.0,
    70,0.0,1.0,
    71,0.0,1.0,
    72,0.0,1.0,
    73,0.0,1.0,
    74,0.0,1.0,
    75,0.02064,1.0,
    76,0.5146,0.86667,
    77,0.18756,0.95,
    78,6e-05,1.0,
    79,0.0,1.0,
    80,0.21298,0.93333,
    81,0.0,1.0,
    82,0.0,1.0,
    83,0.0,1.0,
    84,0.00139,1.0,
    85,0.0,1.0,
    86,0.85297,0.875,
    87,0.0,1.0,
    88,0.0,1.0,
    89,1.45164,0.875,
    90,0.0,1.0,
    91,0.05329,0.92857,
    92,0.55506,0.93333,
    93,0.42187,0.92857,
    94,0.0,1.0,
    95,0.0,1.0,
    96,0.0,1.0,
    97,0.0,1.0,
    98,0.0,1.0,
    99,0.0,1.0,
    100,0.0,1.0,
    101,0.0,1.0,
    102,0.0,1.0,
    103,0.09194,0.95455,
    104,0.0,1.0,
    105,0.0,1.0,
    106,0.05531,0.95,
    107,0.0,1.0,
    108,0.39621,0.95238,
    109,0.0,1.0,
    110,0.8449,0.95,
    111,0.01258,1.0,
    112,0.0,1.0,
    113,0.0,1.0,
    114,0.0,1.0,
    115,0.00355,1.0,
    116,0.0,1.0,
    117,0.3954,0.94118,
    118,0.00259,1.0,
    119,0.0,1.0,
    120,0.0,1.0,
    121,0.35876,0.95,
    122,0.0,1.0,
    123,0.0,1.


key: _node_content
string_value:
{"id_": "2a2ab504-464e-4f9d-95a1-bde4dd212db9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe58b4d0-cd36-45ff-be66-850bf7bb24fb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "035a3ad415f45b998b3f5560eab8eac6560c06f62d3d5479735f3b1dc21abd96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c6288a9-1e50-4e42-a20a-6b4d8e5c9560", "node_type": "1", "metadata": {}, "hash": "7c623d6051c360588130e60d9b3afe240ef4a0588a94f390124282dff50a7767", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
94,0.0,1.0,
    95,0.0,1.0,
    96,0.0,1.0,
    97,0.0,1.0,
    98,0.0,1.0,
    99,0.0,1.0,
    100,0.0,1.0,
    101,0.0,1.0,
    102,0.0,1.0,
    103,0.09194,0.95455,
    104,0.0,1.0,
    105,0.0,1.0,
    106,0.05531,0.95,
    107,0.0,1.0,
    108,0.39621,0.95238,
    109,0.0,1.0,
    110,0.8449,0.95,
    111,0.01258,1.0,
    112,0.0,1.0,
    113,0.0,1.0,
    114,0.0,1.0,
    115,0.00355,1.0,
    116,0.0,1.0,
    117,0.3954,0.94118,
    118,0.00259,1.0,
    119,0.0,1.0,
    120,0.0,1.0,
    121,0.35876,0.95,
    122,0.0,1.0,
    123,0.0,1.0,
    124,5e-05,1.0,
    125,0.0,1.0,
    126,0.0,1.0,
    127,0.0,1.0,
    128,0.0,1.0,
    129,0.0,1.0,
    130,0.01336,1.0,
    131,0.0,1.0,
    132,0.23362,0.95,
    133,0.00157,1.0,
    134,0.0,1.0,
    135,0.00031,1.0,
    136,0.0,1.0,
    137,0.08313,0.92857,
    138,0.0,1.0,
    139,0.0,1.0,
    140,0.0,1.0,
    141,0.43608,0.95,
    142,0.0,1.0,
    143,0.0,1.0,
    144,0.0,1.0,
    145,2e-05,1.0,
    146,1.20409,0.85714,
    147,0.0,1.0,
    148,0.0,1.0,
    149,0.0,1.0,
    150,0.0,1.0,
    151,0.0,


key: _node_content
string_value:
{"id_": "1c6288a9-1e50-4e42-a20a-6b4d8e5c9560", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a2ab504-464e-4f9d-95a1-bde4dd212db9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "94051f70345f186a997cf74e64145f8940fabb6231674735409c612df5799cf5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35763c02-b468-4b3e-9d56-c7b0fb7ddd40", "node_type": "1", "metadata": {}, "hash": "850c8cec2b98aabd1fc34852e06648089482fa91fd3693e0f61389434564d9e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
95,
    122,0.0,1.0,
    123,0.0,1.0,
    124,5e-05,1.0,
    125,0.0,1.0,
    126,0.0,1.0,
    127,0.0,1.0,
    128,0.0,1.0,
    129,0.0,1.0,
    130,0.01336,1.0,
    131,0.0,1.0,
    132,0.23362,0.95,
    133,0.00157,1.0,
    134,0.0,1.0,
    135,0.00031,1.0,
    136,0.0,1.0,
    137,0.08313,0.92857,
    138,0.0,1.0,
    139,0.0,1.0,
    140,0.0,1.0,
    141,0.43608,0.95,
    142,0.0,1.0,
    143,0.0,1.0,
    144,0.0,1.0,
    145,2e-05,1.0,
    146,1.20409,0.85714,
    147,0.0,1.0,
    148,0.0,1.0,
    149,0.0,1.0,
    150,0.0,1.0,
    151,0.0,1.0,
    152,0.0,1.0,
    153,0.0,1.0,
    154,0.00063,1.0,
    155,0.0,1.0,
    156,0.0,1.0,
    157,0.0,1.0,
    158,6e-05,1.0,
    159,0.0,1.0,
    160,0.0,1.0,
    161,0.0,1.0,
    162,0.0,1.0,
    163,0.0,1.0,
    164,0.0,1.0,
    165,0.0,1.0,
    166,0.0,1.0,
    167,0.0,1.0,
    168,0.0,1.0,
    169,0.0,1.0,
    170,0.0,1.0,
    171,0.0,1.0,
    172,0.0,1.0,
    173,0.0,1.0,
    174,0.00783,1.0,
    175,0.0,1.0,
    176,0.0,1.0,
    177,0.0,1.0,
    178,0.0,1.0,
    179,0.0,1.0,


key: _node_content
string_value:
{"id_": "35763c02-b468-4b3e-9d56-c7b0fb7ddd40", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c6288a9-1e50-4e42-a20a-6b4d8e5c9560", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0fdfe21097dba3d474e21bbaa2cd26524f4ea008096223f5dacb08a5477495fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a92c58a-169f-4ea9-9a90-81399c82e552", "node_type": "1", "metadata": {}, "hash": "3abca7580ccc1db1efdd5ac3ebda231a4ee2ec11aa033ead7cd9e828608435d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0,1.0,
    150,0.0,1.0,
    151,0.0,1.0,
    152,0.0,1.0,
    153,0.0,1.0,
    154,0.00063,1.0,
    155,0.0,1.0,
    156,0.0,1.0,
    157,0.0,1.0,
    158,6e-05,1.0,
    159,0.0,1.0,
    160,0.0,1.0,
    161,0.0,1.0,
    162,0.0,1.0,
    163,0.0,1.0,
    164,0.0,1.0,
    165,0.0,1.0,
    166,0.0,1.0,
    167,0.0,1.0,
    168,0.0,1.0,
    169,0.0,1.0,
    170,0.0,1.0,
    171,0.0,1.0,
    172,0.0,1.0,
    173,0.0,1.0,
    174,0.00783,1.0,
    175,0.0,1.0,
    176,0.0,1.0,
    177,0.0,1.0,
    178,0.0,1.0,
    179,0.0,1.0,
    180,0.0,1.0,
    181,0.0,1.0,
    182,0.00028,1.0,
    183,0.0,1.0,
    184,0.0,1.0,
    185,0.0003,1.0,
    186,0.0,1.0,
    187,0.0,1.0,
    188,0.0,1.0,
    189,0.0,1.0,
    190,0.0,1.0,
    191,0.0,1.0,
    192,0.0,1.0,
    193,0.00013,1.0,
    194,0.86198,0.875,
    195,0.0,1.0,
    196,0.0,1.0,
    197,0.0,1.0,
    198,0.0,1.0,
    199,0.0,1.0,
    200,0.0,1.0,
    201,0.0,1.0,
    202,0.0,1.0,
    203,0.0,1.0,
    204,0.09954,0.95455,
    205,0.0,1.0,
    206,0.0,1.0,
    207,0.0,1.0,


key: _node_content
string_value:
{"id_": "4a92c58a-169f-4ea9-9a90-81399c82e552", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35763c02-b468-4b3e-9d56-c7b0fb7ddd40", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b482a0ca564cf2a04449b7a0fdc74f9a99cadd04578c3be4d2bf4edd05615dc1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f6b0aa3-f124-4e5f-8bf3-cca3b09efc7b", "node_type": "1", "metadata": {}, "hash": "eed758f7d00c8064ee0d09f9e8eea3f2b222476ab1baa84fb1c2e482fd6e71a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.0,
    178,0.0,1.0,
    179,0.0,1.0,
    180,0.0,1.0,
    181,0.0,1.0,
    182,0.00028,1.0,
    183,0.0,1.0,
    184,0.0,1.0,
    185,0.0003,1.0,
    186,0.0,1.0,
    187,0.0,1.0,
    188,0.0,1.0,
    189,0.0,1.0,
    190,0.0,1.0,
    191,0.0,1.0,
    192,0.0,1.0,
    193,0.00013,1.0,
    194,0.86198,0.875,
    195,0.0,1.0,
    196,0.0,1.0,
    197,0.0,1.0,
    198,0.0,1.0,
    199,0.0,1.0,
    200,0.0,1.0,
    201,0.0,1.0,
    202,0.0,1.0,
    203,0.0,1.0,
    204,0.09954,0.95455,
    205,0.0,1.0,
    206,0.0,1.0,
    207,0.0,1.0,
    208,1.9616,0.9375,
    209,0.0,1.0,
    210,0.0,1.0,
    211,0.0,1.0,
    212,0.0,1.0,
    213,0.0,1.0,
    214,0.0,1.0,
    215,0.0,1.0,
    216,0.0,1.0,
    217,0.0,1.0,
    218,0.0,1.0,
    219,0.0,1.0,
    220,0.0,1.0,
    221,0.0,1.0,
    222,0.0,1.0,
    223,0.0,1.0,
    224,0.0,1.0,
    225,0.0,1.0,
    226,0.00174,1.0,
    227,0.0,1.0,
    228,2e-05,1.0,
    229,0.0,1.0,
    230,0.0,1.0,
    231,0.0,1.0,
    232,0.0,1.0,
    233,0.0,1.0,
    234,0.61895,0.95,
    235,0.0,1.0,


key: _node_content
string_value:
{"id_": "3f6b0aa3-f124-4e5f-8bf3-cca3b09efc7b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a92c58a-169f-4ea9-9a90-81399c82e552", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b2f8234ee0eb319024915a56afaababb95f78e1e781a7c1c975661e50b939a87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa2c2533-38f9-424f-83c4-7bd41d735d9f", "node_type": "1", "metadata": {}, "hash": "d1c0e9b66d12512ea9ad4956bb60afcd06739c51d3b46970889de6e8ba986e3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.0,
    206,0.0,1.0,
    207,0.0,1.0,
    208,1.9616,0.9375,
    209,0.0,1.0,
    210,0.0,1.0,
    211,0.0,1.0,
    212,0.0,1.0,
    213,0.0,1.0,
    214,0.0,1.0,
    215,0.0,1.0,
    216,0.0,1.0,
    217,0.0,1.0,
    218,0.0,1.0,
    219,0.0,1.0,
    220,0.0,1.0,
    221,0.0,1.0,
    222,0.0,1.0,
    223,0.0,1.0,
    224,0.0,1.0,
    225,0.0,1.0,
    226,0.00174,1.0,
    227,0.0,1.0,
    228,2e-05,1.0,
    229,0.0,1.0,
    230,0.0,1.0,
    231,0.0,1.0,
    232,0.0,1.0,
    233,0.0,1.0,
    234,0.61895,0.95,
    235,0.0,1.0,
    236,0.0,1.0,
    237,0.0,1.0,
    238,0.0,1.0,
    239,0.54945,0.95,
    240,0.0,1.0,
    241,0.0,1.0,
    242,1.52953,0.9375,
    243,1.19938,0.85714,
    244,0.0,1.0,
    245,0.0,1.0,
    246,0.0,1.0,
    247,0.0,1.0,
    248,8e-05,1.0,
    249,0.0,1.0,
    250,0.0,1.0,
    251,0.0,1.0,
    252,0.0,1.0,
    253,0.0,1.0,
    254,0.0,1.0,
    255,0.0,1.0,
    256,0.0,1.0,
    257,0.0,1.0,
    258,0.0,1.0,
    259,0.0,1.0,
    260,0.0,1.0,
    261,0.0,1.0,
    262,0.0,1.0,
    263,0.0,1.0,


key: _node_content
string_value:
{"id_": "aa2c2533-38f9-424f-83c4-7bd41d735d9f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f6b0aa3-f124-4e5f-8bf3-cca3b09efc7b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3beb31456f7f10f34953065271e5d635c4cb3f73cb3975213378802b3a0ccd90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "55c12b71-14a1-4230-afb5-f4253ad3ba1a", "node_type": "1", "metadata": {}, "hash": "58254f10e49445051b507beea5527e779dce00a0c4271dab712663a523086c9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0,
    234,0.61895,0.95,
    235,0.0,1.0,
    236,0.0,1.0,
    237,0.0,1.0,
    238,0.0,1.0,
    239,0.54945,0.95,
    240,0.0,1.0,
    241,0.0,1.0,
    242,1.52953,0.9375,
    243,1.19938,0.85714,
    244,0.0,1.0,
    245,0.0,1.0,
    246,0.0,1.0,
    247,0.0,1.0,
    248,8e-05,1.0,
    249,0.0,1.0,
    250,0.0,1.0,
    251,0.0,1.0,
    252,0.0,1.0,
    253,0.0,1.0,
    254,0.0,1.0,
    255,0.0,1.0,
    256,0.0,1.0,
    257,0.0,1.0,
    258,0.0,1.0,
    259,0.0,1.0,
    260,0.0,1.0,
    261,0.0,1.0,
    262,0.0,1.0,
    263,0.0,1.0,
    264,0.0,1.0,
    265,0.0,1.0,
    266,0.0,1.0,
    267,0.88984,0.95,
    268,0.0,1.0,
    269,0.0,1.0,
    270,0.0,1.0,
    271,0.0,1.0,
    272,0.0,1.0,
    273,0.0,1.0,
    274,0.0,1.0,
    275,0.00013,1.0,
    276,0.0,1.0,
    277,0.89825,0.92857,
    278,0.0,1.0,
    279,0.00017,1.0,
    280,0.0,1.0,
    281,0.0,1.0,
    282,0.0,1.0,
    283,0.65667,0.95,
    284,0.0,1.0,
    285,0.0,1.0,
    286,0.0,1.0,
    287,0.0,1.0,
    288,0.0,1.0,
    289,0.0,1.0,
    290,0.0,1.0,
    291,0.0,1.0,


key: _node_content
string_value:
{"id_": "55c12b71-14a1-4230-afb5-f4253ad3ba1a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa2c2533-38f9-424f-83c4-7bd41d735d9f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e87c99a1f490fcf5ae721292305633cdd7c7a860d4ce048e07da25b97b1d5b5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0a88234-53c2-4849-9553-32f78d0375f7", "node_type": "1", "metadata": {}, "hash": "c5001b785b733e56ad0bd8015dd80728df1aab6e65fdf650cbbd9d562fc1b2a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.0,
    262,0.0,1.0,
    263,0.0,1.0,
    264,0.0,1.0,
    265,0.0,1.0,
    266,0.0,1.0,
    267,0.88984,0.95,
    268,0.0,1.0,
    269,0.0,1.0,
    270,0.0,1.0,
    271,0.0,1.0,
    272,0.0,1.0,
    273,0.0,1.0,
    274,0.0,1.0,
    275,0.00013,1.0,
    276,0.0,1.0,
    277,0.89825,0.92857,
    278,0.0,1.0,
    279,0.00017,1.0,
    280,0.0,1.0,
    281,0.0,1.0,
    282,0.0,1.0,
    283,0.65667,0.95,
    284,0.0,1.0,
    285,0.0,1.0,
    286,0.0,1.0,
    287,0.0,1.0,
    288,0.0,1.0,
    289,0.0,1.0,
    290,0.0,1.0,
    291,0.0,1.0,
    292,0.28626,0.95238,
    293,0.0,1.0,
    294,0.0,1.0,
    295,0.0,1.0,
    296,0.0,1.0,
    297,0.0,1.0,
    298,0.0,1.0,
    299,0.0,1.0,
    300,0.0,1.0,
    301,0.0,1.0,
    302,0.0,1.0,
    303,0.0,1.0,
    304,0.0,1.0,
    305,0.0,1.0,
    306,0.0,1.0,
    307,0.0,1.0,
    308,0.0,1.0,
    309,0.0,1.0,
    


# Evaluations


Great! We trained a fine-tuned model for function calling. Let's see how it does on our evaluation set for prompts that the drone assistant
should automatically reject.


key: _node_content
string_value:
{"id_": "f0a88234-53c2-4849-9553-32f78d0375f7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55c12b71-14a1-4230-afb5-f4253ad3ba1a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "949225cd7d8f60234dff3742738554d8a2ab0f6025307cfecad70213b108415e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b8d80c9e-8efd-47ac-8509-0c445b16b267", "node_type": "1", "metadata": {}, "hash": "4572b15d11d6be83b84cf49475beadd57f41321143bd73586a91d431e80be08a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.0,
    283,0.65667,0.95,
    284,0.0,1.0,
    285,0.0,1.0,
    286,0.0,1.0,
    287,0.0,1.0,
    288,0.0,1.0,
    289,0.0,1.0,
    290,0.0,1.0,
    291,0.0,1.0,
    292,0.28626,0.95238,
    293,0.0,1.0,
    294,0.0,1.0,
    295,0.0,1.0,
    296,0.0,1.0,
    297,0.0,1.0,
    298,0.0,1.0,
    299,0.0,1.0,
    300,0.0,1.0,
    301,0.0,1.0,
    302,0.0,1.0,
    303,0.0,1.0,
    304,0.0,1.0,
    305,0.0,1.0,
    306,0.0,1.0,
    307,0.0,1.0,
    308,0.0,1.0,
    309,0.0,1.0,
    


# Evaluations


Great! We trained a fine-tuned model for function calling. Let's see how it does on our evaluation set for prompts that the drone assistant
should automatically reject.



```python
ft_model = "ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC"
base_model = "gpt-3.5-turbo"

print(f"\nEvaluating fine-tuned model with challenging prompts: {ft_model}")
eval(
    model=ft_model,
    function_list=modified_function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)

print(f"\nEvaluating base model with challenging prompts: {base_model}")
eval(
    model="gpt-3.5-turbo",
    function_list=function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)
```

    
    Evaluating fine-tuned model with challenging prompts: ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC



<style type="text/css">
#T_9f4fa_row0_col0, #T_9f4fa_row0_col1, #T_9f4fa_row0_col2, #T_9f4fa_row0_col3, #T_9f4fa_row1_col0, #T_9f4fa_row1_col1, #T_9f4fa_row1_col2, #T_9f4fa_row1_col3, #T_9f4fa_row2_col0, #T_9f4fa_row2_col1, #T_9f4fa_row2_col2, #T_9f4fa_row2_col3,


key: _node_content
string_value:
{"id_": "b8d80c9e-8efd-47ac-8509-0c445b16b267", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0a88234-53c2-4849-9553-32f78d0375f7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d91818355339fa63fc5a81c9d06c6ce5d990b9ed469bc0b83403befaa4687e55", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f18182f0-ad4b-43da-9658-05b021f514ae", "node_type": "1", "metadata": {}, "hash": "8ea7dea9e11f1650c6359a76e4a4d12d28e31739f73514d40d2b6bf2d9bb84ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 72581, "end_char_idx": 74299, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
ft_model = "ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC"
base_model = "gpt-3.5-turbo"

print(f"\nEvaluating fine-tuned model with challenging prompts: {ft_model}")
eval(
    model=ft_model,
    function_list=modified_function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)

print(f"\nEvaluating base model with challenging prompts: {base_model}")
eval(
    model="gpt-3.5-turbo",
    function_list=function_list,
    system_prompt=DRONE_SYSTEM_PROMPT,
    prompts_to_expected_tool_name=challenging_prompts_to_expected,
)
```

    
    Evaluating fine-tuned model with challenging prompts: ft:gpt-3.5-turbo-0125:openai-gtm:drone:9atiPjeC



<style type="text/css">
#T_9f4fa_row0_col0, #T_9f4fa_row0_col1, #T_9f4fa_row0_col2, #T_9f4fa_row0_col3, #T_9f4fa_row1_col0, #T_9f4fa_row1_col1, #T_9f4fa_row1_col2, #T_9f4fa_row1_col3, #T_9f4fa_row2_col0, #T_9f4fa_row2_col1, #T_9f4fa_row2_col2, #T_9f4fa_row2_col3, #T_9f4fa_row3_col0, #T_9f4fa_row3_col1, #T_9f4fa_row3_col2, #T_9f4fa_row3_col3, #T_9f4fa_row4_col0, #T_9f4fa_row4_col1, #T_9f4fa_row4_col2, #T_9f4fa_row4_col3, #T_9f4fa_row5_col0, #T_9f4fa_row5_col1, #T_9f4fa_row5_col2, #T_9f4fa_row5_col3, #T_9f4fa_row6_col0, #T_9f4fa_row6_col1, #T_9f4fa_row6_col2, #T_9f4fa_row6_col3, #T_9f4fa_row7_col0, #T_9f4fa_row7_col1, #T_9f4fa_row7_col2, #T_9f4fa_row7_col3, #T_9f4fa_row8_col0, #T_9f4fa_row8_col1, #T_9f4fa_row8_col2, #T_9f4fa_row8_col3, #T_9f4fa_row9_col0, #T_9f4fa_row9_col1, #T_9f4fa_row9_col2, #T_9f4fa_row9_col3 {
  background-color: white;
  color: black;
}
</style>
<table id="T_9f4fa">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_9f4fa_level0_col0"


key: _node_content
string_value:
{"id_": "f18182f0-ad4b-43da-9658-05b021f514ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b8d80c9e-8efd-47ac-8509-0c445b16b267", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "97f5af8ec4534df0cb17831b021f294d3dc2c983f6e9490432201c53b44b595c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f1e140b-3289-424a-bef7-630810626a92", "node_type": "1", "metadata": {}, "hash": "c6334e47a75453334883245f69b5919fa19459a156cfdd35a5ccb8bde0160d83", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 73586, "end_char_idx": 75153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#T_9f4fa_row3_col1, #T_9f4fa_row3_col2, #T_9f4fa_row3_col3, #T_9f4fa_row4_col0, #T_9f4fa_row4_col1, #T_9f4fa_row4_col2, #T_9f4fa_row4_col3, #T_9f4fa_row5_col0, #T_9f4fa_row5_col1, #T_9f4fa_row5_col2, #T_9f4fa_row5_col3, #T_9f4fa_row6_col0, #T_9f4fa_row6_col1, #T_9f4fa_row6_col2, #T_9f4fa_row6_col3, #T_9f4fa_row7_col0, #T_9f4fa_row7_col1, #T_9f4fa_row7_col2, #T_9f4fa_row7_col3, #T_9f4fa_row8_col0, #T_9f4fa_row8_col1, #T_9f4fa_row8_col2, #T_9f4fa_row8_col3, #T_9f4fa_row9_col0, #T_9f4fa_row9_col1, #T_9f4fa_row9_col2, #T_9f4fa_row9_col3 {
  background-color: white;
  color: black;
}
</style>
<table id="T_9f4fa">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_9f4fa_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_9f4fa_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_9f4fa_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_9f4fa_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_9f4fa_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_9f4fa_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_9f4fa_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_9f4fa_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_9f4fa_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_9f4fa_row1_col0" class="data


key: _node_content
string_value:
{"id_": "7f1e140b-3289-424a-bef7-630810626a92", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f18182f0-ad4b-43da-9658-05b021f514ae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ba92c8e76ae0b4a4ea453013dad79f1e5ccf48464aaeb7f6f24b079e36c2edfc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b7a8129-ebd5-4bec-bb80-e381ec21c8c6", "node_type": "1", "metadata": {}, "hash": "163b6b3083ebebefe8687a37fd5bc4408a3af6638730fb1f5dac4deb21acd8a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 74271, "end_char_idx": 75988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<th id="T_9f4fa_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_9f4fa_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_9f4fa_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_9f4fa_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_9f4fa_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_9f4fa_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_9f4fa_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_9f4fa_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_9f4fa_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_9f4fa_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_9f4fa_row1_col1" class="data row1 col1" >reject_request</td>
      <td id="T_9f4fa_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_9f4fa_row1_col3" class="data row1 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_9f4fa_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_9f4fa_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_9f4fa_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_9f4fa_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_9f4fa_row3_col0" class="data row3 col0"


key: _node_content
string_value:
{"id_": "7b7a8129-ebd5-4bec-bb80-e381ec21c8c6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f1e140b-3289-424a-bef7-630810626a92", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a75ef8b8b79344029fe265371bfc95be904837d86a62382a4c4a493f0e49b0f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03bc1cd2-df24-496f-800d-071007b7e37e", "node_type": "1", "metadata": {}, "hash": "3efd044562dbd2831d3bc288df22c5d428314b8bb8ef2f5e1610b69427a9c0f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 75115, "end_char_idx": 76827, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_9f4fa_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_9f4fa_row1_col1" class="data row1 col1" >reject_request</td>
      <td id="T_9f4fa_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_9f4fa_row1_col3" class="data row1 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_9f4fa_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_9f4fa_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_9f4fa_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_9f4fa_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_9f4fa_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_9f4fa_row3_col1" class="data row3 col1" >reject_request</td>
      <td id="T_9f4fa_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_9f4fa_row3_col3" class="data row3 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_9f4fa_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_9f4fa_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_9f4fa_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_9f4fa_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_9f4fa_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>


key: _node_content
string_value:
{"id_": "03bc1cd2-df24-496f-800d-071007b7e37e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b7a8129-ebd5-4bec-bb80-e381ec21c8c6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "296342d307877065c1d0a0bc774f5b47b07b211eadc24e6eae4934ab267a9d74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad51bb71-b2f2-4d95-99a3-0cc86175fddf", "node_type": "1", "metadata": {}, "hash": "7002d689ade60da703eaa5fe4ce57c97ceb5790db94612ee4565dfdf2a02bb36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 75939, "end_char_idx": 77629, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td id="T_9f4fa_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_9f4fa_row3_col1" class="data row3 col1" >reject_request</td>
      <td id="T_9f4fa_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_9f4fa_row3_col3" class="data row3 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_9f4fa_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_9f4fa_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_9f4fa_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_9f4fa_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_9f4fa_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_9f4fa_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_9f4fa_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_9f4fa_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_9f4fa_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_9f4fa_row6_col1" class="data row6 col1" >reject_request</td>
      <td id="T_9f4fa_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_9f4fa_row6_col3" class="data row6 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_9f4fa_row7_col0" class="data row7 col0" >Detect a person</td>


key: _node_content
string_value:
{"id_": "ad51bb71-b2f2-4d95-99a3-0cc86175fddf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03bc1cd2-df24-496f-800d-071007b7e37e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "fc245240bace8b990ae72da5752ba473236ee002b68dfca0df2cfe8a71f996d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d56038f2-d9ef-4d2a-8d9b-5592570bfdb7", "node_type": "1", "metadata": {}, "hash": "591d777939cb095f9eadc19389fba03b34a86902bdab92250f087f00a51261f0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 76769, "end_char_idx": 78478, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_9f4fa_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_9f4fa_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_9f4fa_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_9f4fa_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_9f4fa_row6_col1" class="data row6 col1" >reject_request</td>
      <td id="T_9f4fa_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_9f4fa_row6_col3" class="data row6 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_9f4fa_row7_col0" class="data row7 col0" >Detect a person</td>
      <td id="T_9f4fa_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_9f4fa_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_9f4fa_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_9f4fa_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_9f4fa_row8_col1" class="data row8 col1" >reject_request</td>
      <td id="T_9f4fa_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_9f4fa_row8_col3" class="data row8 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_9f4fa_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_9f4fa_row9_col1"


key: _node_content
string_value:
{"id_": "d56038f2-d9ef-4d2a-8d9b-5592570bfdb7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad51bb71-b2f2-4d95-99a3-0cc86175fddf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "93d5cb410c05fc2ed5bbe4c1034d4d9050018eb29ae396541c424473525b1ae9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e27aa4b-6ea4-4b23-80cf-103287c62b72", "node_type": "1", "metadata": {}, "hash": "3febad17fd59a387dddc4a41a828419ba64bc0fdbe6cb51a9f7f2b57004ea801", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 77597, "end_char_idx": 79350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
row7 col0" >Detect a person</td>
      <td id="T_9f4fa_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_9f4fa_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_9f4fa_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_9f4fa_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_9f4fa_row8_col1" class="data row8 col1" >reject_request</td>
      <td id="T_9f4fa_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_9f4fa_row8_col3" class="data row8 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_9f4fa_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_9f4fa_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_9f4fa_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_9f4fa_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_9f4fa_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 10 out of 10 (100.00%)
    Average latency per request: 3519.17 ms
    Average tokens used per request: 457.20
    
    Evaluating base model with challenging prompts: gpt-3.5-turbo



<style type="text/css">
#T_85118_row0_col0, #T_85118_row0_col1, #T_85118_row0_col2, #T_85118_row0_col3, #T_85118_row2_col0, #T_85118_row2_col1, #T_85118_row2_col2, #T_85118_row2_col3, #T_85118_row4_col0, #T_85118_row4_col1, #T_85118_row4_col2, #T_85118_row4_col3, #T_85118_row5_col0, #T_85118_row5_col1, #T_85118_row5_col2, #T_85118_row5_col3, #T_85118_row7_col0, #T_85118_row7_col1, #T_85118_row7_col2, #T_85118_row7_col3, #T_85118_row9_col0,


key: _node_content
string_value:
{"id_": "4e27aa4b-6ea4-4b23-80cf-103287c62b72", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d56038f2-d9ef-4d2a-8d9b-5592570bfdb7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "55b99f8cfb47e7113dbb3b921843a9eaed7b57409317724ce57c2a8924a1680f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7892a84-a4de-4d5a-aba6-63bccf87757b", "node_type": "1", "metadata": {}, "hash": "e0d781d9e2ed2297906de2f5233e633552f92d3d16f8b77f98e3b9d2daab45ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 78411, "end_char_idx": 80216, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
on humidity levels around you</td>
      <td id="T_9f4fa_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_9f4fa_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_9f4fa_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 10 out of 10 (100.00%)
    Average latency per request: 3519.17 ms
    Average tokens used per request: 457.20
    
    Evaluating base model with challenging prompts: gpt-3.5-turbo



<style type="text/css">
#T_85118_row0_col0, #T_85118_row0_col1, #T_85118_row0_col2, #T_85118_row0_col3, #T_85118_row2_col0, #T_85118_row2_col1, #T_85118_row2_col2, #T_85118_row2_col3, #T_85118_row4_col0, #T_85118_row4_col1, #T_85118_row4_col2, #T_85118_row4_col3, #T_85118_row5_col0, #T_85118_row5_col1, #T_85118_row5_col2, #T_85118_row5_col3, #T_85118_row7_col0, #T_85118_row7_col1, #T_85118_row7_col2, #T_85118_row7_col3, #T_85118_row9_col0, #T_85118_row9_col1, #T_85118_row9_col2, #T_85118_row9_col3 {
  background-color: white;
  color: black;
}
#T_85118_row1_col0, #T_85118_row1_col1, #T_85118_row1_col2, #T_85118_row1_col3, #T_85118_row3_col0, #T_85118_row3_col1, #T_85118_row3_col2, #T_85118_row3_col3, #T_85118_row6_col0, #T_85118_row6_col1, #T_85118_row6_col2, #T_85118_row6_col3, #T_85118_row8_col0, #T_85118_row8_col1, #T_85118_row8_col2, #T_85118_row8_col3 {
  background-color: red;
  color: black;
}
</style>
<table id="T_85118">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_85118_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_85118_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_85118_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_85118_level0_col3" class="col_heading level0


key: _node_content
string_value:
{"id_": "b7892a84-a4de-4d5a-aba6-63bccf87757b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e27aa4b-6ea4-4b23-80cf-103287c62b72", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a5a4c722182aaf4660913def8bbd075a81a17a306e4daed96b7f5dfb044a95a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c6ceb26-62f5-4e03-9447-4f6701acefec", "node_type": "1", "metadata": {}, "hash": "49815a8abdbac9e1e4b24ad2dcf871bd86cd118220fe3bad9f2408c051a269e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 79311, "end_char_idx": 81096, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#T_85118_row7_col3, #T_85118_row9_col0, #T_85118_row9_col1, #T_85118_row9_col2, #T_85118_row9_col3 {
  background-color: white;
  color: black;
}
#T_85118_row1_col0, #T_85118_row1_col1, #T_85118_row1_col2, #T_85118_row1_col3, #T_85118_row3_col0, #T_85118_row3_col1, #T_85118_row3_col2, #T_85118_row3_col3, #T_85118_row6_col0, #T_85118_row6_col1, #T_85118_row6_col2, #T_85118_row6_col3, #T_85118_row8_col0, #T_85118_row8_col1, #T_85118_row8_col2, #T_85118_row8_col3 {
  background-color: red;
  color: black;
}
</style>
<table id="T_85118">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_85118_level0_col0" class="col_heading level0 col0" >Prompt</th>
      <th id="T_85118_level0_col1" class="col_heading level0 col1" >Actual</th>
      <th id="T_85118_level0_col2" class="col_heading level0 col2" >Expected</th>
      <th id="T_85118_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_85118_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_85118_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_85118_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_85118_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_85118_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_85118_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_85118_row1_col1" class="data row1 col1" >set_follow_me_mode</td>
      <td id="T_85118_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_85118_row1_col3" class="data row1 col3" >No</td>
    </tr>
    <tr>


key: _node_content
string_value:
{"id_": "5c6ceb26-62f5-4e03-9447-4f6701acefec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7892a84-a4de-4d5a-aba6-63bccf87757b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0ea2002f199cee741a2160c2dfc7dcc0b89d55be03bfcf7b5f0b1df878e6f9dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "939598f8-feca-466a-80b3-7cb6e64e0163", "node_type": "1", "metadata": {}, "hash": "2e75031371b36fc5728f0f7436c193ac59f385d71b349dfbcf147b9ea961c296", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 80162, "end_char_idx": 81966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<th id="T_85118_level0_col3" class="col_heading level0 col3" >Match</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_85118_level0_row0" class="row_heading level0 row0" >0</th>
      <td id="T_85118_row0_col0" class="data row0 col0" >Play pre-recorded audio message</td>
      <td id="T_85118_row0_col1" class="data row0 col1" >reject_request</td>
      <td id="T_85118_row0_col2" class="data row0 col2" >reject_request</td>
      <td id="T_85118_row0_col3" class="data row0 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row1" class="row_heading level0 row1" >1</th>
      <td id="T_85118_row1_col0" class="data row1 col0" >Initiate following on social media</td>
      <td id="T_85118_row1_col1" class="data row1 col1" >set_follow_me_mode</td>
      <td id="T_85118_row1_col2" class="data row1 col2" >reject_request</td>
      <td id="T_85118_row1_col3" class="data row1 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_85118_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_85118_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_85118_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_85118_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_85118_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_85118_row3_col1" class="data row3 col1" >set_obstacle_avoidance</td>
      <td id="T_85118_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_85118_row3_col3" class="data row3 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row4" class="row_heading


key: _node_content
string_value:
{"id_": "939598f8-feca-466a-80b3-7cb6e64e0163", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c6ceb26-62f5-4e03-9447-4f6701acefec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8f0b117d2221a29653165fa936d90d29774b025844fc2bd9a8717b91d56acdf5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "068b65bc-6955-4dba-bbaa-469a049b9a6f", "node_type": "1", "metadata": {}, "hash": "ce148d400f125e4b5348defec639f03280131ee7846df36ea6e3f6adc264d5cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 81046, "end_char_idx": 82810, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="data row1 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row2" class="row_heading level0 row2" >2</th>
      <td id="T_85118_row2_col0" class="data row2 col0" >Scan environment for heat signatures</td>
      <td id="T_85118_row2_col1" class="data row2 col1" >reject_request</td>
      <td id="T_85118_row2_col2" class="data row2 col2" >reject_request</td>
      <td id="T_85118_row2_col3" class="data row2 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row3" class="row_heading level0 row3" >3</th>
      <td id="T_85118_row3_col0" class="data row3 col0" >Bump into obstacles</td>
      <td id="T_85118_row3_col1" class="data row3 col1" >set_obstacle_avoidance</td>
      <td id="T_85118_row3_col2" class="data row3 col2" >reject_request</td>
      <td id="T_85118_row3_col3" class="data row3 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_85118_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_85118_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_85118_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_85118_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_85118_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_85118_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_85118_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_85118_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row6" class="row_heading level0 row6" >6</th>
      <td


key: _node_content
string_value:
{"id_": "068b65bc-6955-4dba-bbaa-469a049b9a6f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "939598f8-feca-466a-80b3-7cb6e64e0163", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6b1ffb30edf83ac643757dc2ceb50b0751a2b97a7dd99b611583beb4062ad009", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d32c96d8-283e-4d8f-b8a4-a9feba51ce68", "node_type": "1", "metadata": {}, "hash": "47462759ce1620ef752c7af185c61b37496e6d828158b6e175147a7cac641100", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 81898, "end_char_idx": 83679, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</tr>
    <tr>
      <th id="T_85118_level0_row4" class="row_heading level0 row4" >4</th>
      <td id="T_85118_row4_col0" class="data row4 col0" >Change drone's paint job color</td>
      <td id="T_85118_row4_col1" class="data row4 col1" >reject_request</td>
      <td id="T_85118_row4_col2" class="data row4 col2" >reject_request</td>
      <td id="T_85118_row4_col3" class="data row4 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row5" class="row_heading level0 row5" >5</th>
      <td id="T_85118_row5_col0" class="data row5 col0" >Coordinate with nearby drones</td>
      <td id="T_85118_row5_col1" class="data row5 col1" >reject_request</td>
      <td id="T_85118_row5_col2" class="data row5 col2" >reject_request</td>
      <td id="T_85118_row5_col3" class="data row5 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row6" class="row_heading level0 row6" >6</th>
      <td id="T_85118_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_85118_row6_col1" class="data row6 col1" >set_drone_speed</td>
      <td id="T_85118_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_85118_row6_col3" class="data row6 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_85118_row7_col0" class="data row7 col0" >Detect a person</td>
      <td id="T_85118_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_85118_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_85118_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_85118_row8_col0" class="data row8 col0" >Please enable night


key: _node_content
string_value:
{"id_": "d32c96d8-283e-4d8f-b8a4-a9feba51ce68", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "068b65bc-6955-4dba-bbaa-469a049b9a6f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "450c62be3d3ea611c68c3bc9e72b08c14aebc5db4dac1ab9ae163892b7f493b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7fe4cd83-879a-4187-b9ed-089dea947619", "node_type": "1", "metadata": {}, "hash": "6624688739c4b350e191f78fe2169f415381698432ea7e950aca6298af023ee8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 82761, "end_char_idx": 84636, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class="row_heading level0 row6" >6</th>
      <td id="T_85118_row6_col0" class="data row6 col0" >Change speed to negative 120 km/h</td>
      <td id="T_85118_row6_col1" class="data row6 col1" >set_drone_speed</td>
      <td id="T_85118_row6_col2" class="data row6 col2" >reject_request</td>
      <td id="T_85118_row6_col3" class="data row6 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row7" class="row_heading level0 row7" >7</th>
      <td id="T_85118_row7_col0" class="data row7 col0" >Detect a person</td>
      <td id="T_85118_row7_col1" class="data row7 col1" >reject_request</td>
      <td id="T_85118_row7_col2" class="data row7 col2" >reject_request</td>
      <td id="T_85118_row7_col3" class="data row7 col3" >Yes</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row8" class="row_heading level0 row8" >8</th>
      <td id="T_85118_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_85118_row8_col1" class="data row8 col1" >set_drone_lighting</td>
      <td id="T_85118_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_85118_row8_col3" class="data row8 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_85118_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_85118_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_85118_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_85118_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 6 out of 10 (60.00%)
    Average latency per request: 647.58 ms
    Average tokens used per request: 791.90


Great! While the original model only rejected 60%, the fine tuned model rejected 100% requests and used less tokens to do so.


### Conclusion


key: _node_content
string_value:
{"id_": "7fe4cd83-879a-4187-b9ed-089dea947619", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e2a7ce-e4b6-45ef-b28e-50b87a57779c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "660de8d98c1ff3119943c9886daca4850fedcc37b81b494ed3a3eb27f7f4ba48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d32c96d8-283e-4d8f-b8a4-a9feba51ce68", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Fine_tuning_for_function_calling.ipynb", "title": "Fine tuning with function-calling", "tokens": 26666, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a2a55263a23e56ebe31101c92c5c0fa719aa954c8ef69c895984daa7109f3343", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 83595, "end_char_idx": 84756, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>8</th>
      <td id="T_85118_row8_col0" class="data row8 col0" >Please enable night vision</td>
      <td id="T_85118_row8_col1" class="data row8 col1" >set_drone_lighting</td>
      <td id="T_85118_row8_col2" class="data row8 col2" >reject_request</td>
      <td id="T_85118_row8_col3" class="data row8 col3" >No</td>
    </tr>
    <tr>
      <th id="T_85118_level0_row9" class="row_heading level0 row9" >9</th>
      <td id="T_85118_row9_col0" class="data row9 col0" >Report on humidity levels around you</td>
      <td id="T_85118_row9_col1" class="data row9 col1" >reject_request</td>
      <td id="T_85118_row9_col2" class="data row9 col2" >reject_request</td>
      <td id="T_85118_row9_col3" class="data row9 col3" >Yes</td>
    </tr>
  </tbody>
</table>



    Number of matches: 6 out of 10 (60.00%)
    Average latency per request: 647.58 ms
    Average tokens used per request: 791.90


Great! While the original model only rejected 60%, the fine tuned model rejected 100% requests and used less tokens to do so.


### Conclusion


Congratulations! You are now ready to fine tune your model for function calling. We can't wait to see what you build.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb


key: _node_content
string_value:
{"id_": "00297e7b-e521-4d1b-9bda-1957d0adfd21", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "00f6869e-60ff-4e5b-871f-4f470a7e0a6f", "node_type": "1", "metadata": {}, "hash": "5940647d306d318951f8c7c64d53036d63c49c1f1b0a30f68bd54105a4f89b24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2471, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Function-calling with an OpenAPI specification


Much of the internet is powered by RESTful APIs. Giving GPT the ability to call them opens up a world of possibilities. This notebook demonstrates how GPTs can be used to intelligently call APIs. It leverages OpenAPI specifications and chained function calls.

The [OpenAPI Specification (OAS)](https://swagger.io/specification/) is a universally accepted standard for describing the details of RESTful APIs in a format that machines can read and interpret. It enables both humans and computers to understand the capabilities of a service, and it can be leveraged to show GPT how to call APIs.

This notebook is divided into two main sections:

1. How to convert a sample OpenAPI specification into a list of function definitions for the chat completions API.
2. How to use the chat completions API to intelligently invoke these functions based on user instructions.

We recommend familiariazing yourself with [function-calling](./How_to_call_functions_with_chat_models.ipynb) before proceding.



```python
!pip install -q jsonref # for resolving $ref's in the OpenAPI spec
!pip install -q openai
```

    [33mDEPRECATION: textract 1.6.5 has a non-standard dependency specifier extract-msg<=0.29.*. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of textract or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063[0m[33m
    [0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m23.3.1[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m
    [33mDEPRECATION: textract 1.6.5 has a non-standard dependency specifier extract-msg<=0.29.*. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of textract or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063[0m[33m
    [0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.2.1[0m[39;49m -> [0m[32;49m23.3.1[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m


key: _node_content
string_value:
{"id_": "00f6869e-60ff-4e5b-871f-4f470a7e0a6f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00297e7b-e521-4d1b-9bda-1957d0adfd21", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "cec18714b97655f44bf6736f1713f19680063e68d7e64943039e02e426e9fcd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c452a4b-a10d-4fcb-b6f1-4872160b7ace", "node_type": "1", "metadata": {}, "hash": "7fe3a918b35f7dbe0ac3bfce3ef55995f92809263c3f7fb58dfb0add6152b0ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2475, "end_char_idx": 5600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import os
import json
import jsonref
from openai import OpenAI
import requests
from pprint import pp

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

## How to convert an OpenAPI specification into function definitions


The example OpenAPI spec we use here was created using `gpt-4`. We will transform this sample spec into a set of function definitions that can be supplied to the chat completion API. The model, based on the provided user instructions, generates a JSON object containing the necessary arguments to call these functions.

Before we proceed, let's inspect this generated spec. OpenAPI specs include details about the API's endpoints, the operations they support, the parameters they accept, the requests they can handle, and the responses they return. The spec is defined in JSON format.

The endpoints in the spec include operations for:

- Listing all events
- Creating a new event
- Retrieving an event by ID
- Deleting an event by ID
- Updating an event name by ID

Each operation in the spec has an `operationId`, which we will use as the function name when we parse the spec into function specifications. The spec also includes schemas that define the data types and structures of the parameters for each operation.

You can see the schema here:



```python
with open('./data/example_events_openapi.json', 'r') as f:
    openapi_spec = jsonref.loads(f.read()) # it's important to load with jsonref, as explained below

display(openapi_spec)
```


    {'openapi': '3.0.0',
     'info': {'version': '1.0.0',
      'title': 'Event Management API',
      'description': 'An API for managing event data'},
     'paths': {'/events': {'get': {'summary': 'List all events',
        'operationId': 'listEvents',
        'responses': {'200': {'description': 'A list of events',
          'content': {'application/json': {'schema': {'type': 'array',
             'items': {'type': 'object',
              'properties': {'id': {'type': 'string'},
               'name': {'type': 'string'},
               'date': {'type': 'string', 'format': 'date-time'},
               'location': {'type': 'string'}},
              'required': ['name', 'date', 'location']}}}}}}},
       'post': {'summary': 'Create a new event',
        'operationId': 'createEvent',
        'requestBody': {'required': True,
         'content': {'application/json': {'schema': {'type': 'object',
            'properties': {'id': {'type': 'string'},
             'name': {'type': 'string'},
             'date': {'type': 'string', 'format': 'date-time'},
             'location': {'type': 'string'}},
            'required': ['name', 'date', 'location']}}}},
        'responses': {'201': {'description': 'The event was created',
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}}},


key: _node_content
string_value:
{"id_": "9c452a4b-a10d-4fcb-b6f1-4872160b7ace", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "00f6869e-60ff-4e5b-871f-4f470a7e0a6f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "49a26104442c442d2bd784ab9f8f453f51207fffe18da3296154f05cfdfb5b4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f01a2503-b9d1-40f1-8683-2135ba253532", "node_type": "1", "metadata": {}, "hash": "49b31341abaf4efbd8668b1f266c51f336105eec26858880eb670048ffaea364", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4123, "end_char_idx": 6984, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'description': 'An API for managing event data'},
     'paths': {'/events': {'get': {'summary': 'List all events',
        'operationId': 'listEvents',
        'responses': {'200': {'description': 'A list of events',
          'content': {'application/json': {'schema': {'type': 'array',
             'items': {'type': 'object',
              'properties': {'id': {'type': 'string'},
               'name': {'type': 'string'},
               'date': {'type': 'string', 'format': 'date-time'},
               'location': {'type': 'string'}},
              'required': ['name', 'date', 'location']}}}}}}},
       'post': {'summary': 'Create a new event',
        'operationId': 'createEvent',
        'requestBody': {'required': True,
         'content': {'application/json': {'schema': {'type': 'object',
            'properties': {'id': {'type': 'string'},
             'name': {'type': 'string'},
             'date': {'type': 'string', 'format': 'date-time'},
             'location': {'type': 'string'}},
            'required': ['name', 'date', 'location']}}}},
        'responses': {'201': {'description': 'The event was created',
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}}},
      '/events/{id}': {'get': {'summary': 'Retrieve an event by ID',
        'operationId': 'getEventById',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'responses': {'200': {'description': 'The event',
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}},
       'delete': {'summary': 'Delete an event by ID',
        'operationId': 'deleteEvent',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'responses': {'204': {'description': 'The event was deleted'}}},
       'patch': {'summary': "Update an event's details by ID",
        'operationId': 'updateEventDetails',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'requestBody': {'required': True,
         'content': {'application/json': {'schema': {'type': 'object',
            'properties': {'name': {'type': 'string'},
             'date': {'type': 'string',


key: _node_content
string_value:
{"id_": "f01a2503-b9d1-40f1-8683-2135ba253532", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c452a4b-a10d-4fcb-b6f1-4872160b7ace", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "fe18844165a6711207d221059f3ad1b162dbe20330001be6ac073093146d8f3b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4411bb46-a01d-4411-8958-7605ca2edb1a", "node_type": "1", "metadata": {}, "hash": "923135d06eb2709bc8a165253efaa8748568e980def448d1926a3b7e7ca59de1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5552, "end_char_idx": 7837, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'required': ['name', 'date', 'location']}}}}}}},
      '/events/{id}': {'get': {'summary': 'Retrieve an event by ID',
        'operationId': 'getEventById',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'responses': {'200': {'description': 'The event',
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}},
       'delete': {'summary': 'Delete an event by ID',
        'operationId': 'deleteEvent',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'responses': {'204': {'description': 'The event was deleted'}}},
       'patch': {'summary': "Update an event's details by ID",
        'operationId': 'updateEventDetails',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'requestBody': {'required': True,
         'content': {'application/json': {'schema': {'type': 'object',
            'properties': {'name': {'type': 'string'},
             'date': {'type': 'string', 'format': 'date-time'},
             'location': {'type': 'string'}},
            'required': ['name', 'date', 'location']}}}},
        'responses': {'200': {'description': "The event's details were updated",
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}}}},
     'components': {'schemas': {'Event': {'type': 'object',
        'properties': {'id': {'type': 'string'},
         'name': {'type': 'string'},
         'date': {'type': 'string', 'format': 'date-time'},
         'location': {'type': 'string'}},
        'required': ['name', 'date', 'location']}}}}


key: _node_content
string_value:
{"id_": "4411bb46-a01d-4411-8958-7605ca2edb1a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f01a2503-b9d1-40f1-8683-2135ba253532", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c934b44bd90c7b0f4763b7d83d85baab3961ce71aeecddd7de0a4d08604161bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "062d4c9c-61fe-46dd-9a8e-2c7af4618d6e", "node_type": "1", "metadata": {}, "hash": "11c807733f15d26f759bfc485900142aefa6f699790cce8137acae67d51562b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6380, "end_char_idx": 9504, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'responses': {'204': {'description': 'The event was deleted'}}},
       'patch': {'summary': "Update an event's details by ID",
        'operationId': 'updateEventDetails',
        'parameters': [{'name': 'id',
          'in': 'path',
          'required': True,
          'schema': {'type': 'string'}}],
        'requestBody': {'required': True,
         'content': {'application/json': {'schema': {'type': 'object',
            'properties': {'name': {'type': 'string'},
             'date': {'type': 'string', 'format': 'date-time'},
             'location': {'type': 'string'}},
            'required': ['name', 'date', 'location']}}}},
        'responses': {'200': {'description': "The event's details were updated",
          'content': {'application/json': {'schema': {'type': 'object',
             'properties': {'id': {'type': 'string'},
              'name': {'type': 'string'},
              'date': {'type': 'string', 'format': 'date-time'},
              'location': {'type': 'string'}},
             'required': ['name', 'date', 'location']}}}}}}}},
     'components': {'schemas': {'Event': {'type': 'object',
        'properties': {'id': {'type': 'string'},
         'name': {'type': 'string'},
         'date': {'type': 'string', 'format': 'date-time'},
         'location': {'type': 'string'}},
        'required': ['name', 'date', 'location']}}}}


Now that we have a good understanding of the OpenAPI spec, we can proceed to parse it into function specifications.

We can write a simple `openapi_to_functions` function to generate a list of definitions, where each function is represented as a dictionary containing the following keys:

- `name`: This corresponds to the operation identifier of the API endpoint as defined in the OpenAPI specification.
- `description`: This is a brief description or summary of the function, providing an overview of what the function does.
- `parameters`: This is a schema that defines the expected input parameters for the function. It provides information about the type of each parameter, whether it is required or optional, and other related details.

For each of the endpoints defined in the schema, we need to do the following:

1. **Resolve JSON references**: In an OpenAPI specification, it's common to use JSON references (also known as $ref) to avoid duplication. These references point to definitions that are used in multiple places. For example, if multiple API endpoints return the same object structure, that structure can be defined once and then referenced wherever it's needed. We need to resolve and replace these references with the content they point to.

2. **Extract a name for the functions:** We will simply use the operationId as the function name. Alternatively, we could use the endpoint path and operation as the function name.

3. **Extract a description and parameters:** We will iterate through the `description`, `summary`, `requestBody` and `parameters` fields to populate the function's description and parameters.

Here's the implementation:


key: _node_content
string_value:
{"id_": "062d4c9c-61fe-46dd-9a8e-2c7af4618d6e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4411bb46-a01d-4411-8958-7605ca2edb1a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "652122d15b7bf53689ce290c7549024a17dd2bcf8c9aa66874aab92ba1e3b606", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b74e8ffc-3e08-4c4e-8a7b-b2485c568ec4", "node_type": "1", "metadata": {}, "hash": "6905fa756180a30e798d0f8e47e18505332d194fd9f585f61c18c56eda6c2af7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7605, "end_char_idx": 10963, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'properties': {'id': {'type': 'string'},
         'name': {'type': 'string'},
         'date': {'type': 'string', 'format': 'date-time'},
         'location': {'type': 'string'}},
        'required': ['name', 'date', 'location']}}}}


Now that we have a good understanding of the OpenAPI spec, we can proceed to parse it into function specifications.

We can write a simple `openapi_to_functions` function to generate a list of definitions, where each function is represented as a dictionary containing the following keys:

- `name`: This corresponds to the operation identifier of the API endpoint as defined in the OpenAPI specification.
- `description`: This is a brief description or summary of the function, providing an overview of what the function does.
- `parameters`: This is a schema that defines the expected input parameters for the function. It provides information about the type of each parameter, whether it is required or optional, and other related details.

For each of the endpoints defined in the schema, we need to do the following:

1. **Resolve JSON references**: In an OpenAPI specification, it's common to use JSON references (also known as $ref) to avoid duplication. These references point to definitions that are used in multiple places. For example, if multiple API endpoints return the same object structure, that structure can be defined once and then referenced wherever it's needed. We need to resolve and replace these references with the content they point to.

2. **Extract a name for the functions:** We will simply use the operationId as the function name. Alternatively, we could use the endpoint path and operation as the function name.

3. **Extract a description and parameters:** We will iterate through the `description`, `summary`, `requestBody` and `parameters` fields to populate the function's description and parameters.

Here's the implementation:



```python
def openapi_to_functions(openapi_spec):
    functions = []

    for path, methods in openapi_spec["paths"].items():
        for method, spec_with_ref in methods.items():
            # 1. Resolve JSON references.
            spec = jsonref.replace_refs(spec_with_ref)

            # 2. Extract a name for the functions.
            function_name = spec.get("operationId")

            # 3. Extract a description and parameters.
            desc = spec.get("description") or spec.get("summary", "")

            schema = {"type": "object", "properties": {}}

            req_body = (
                spec.get("requestBody", {})
                .get("content", {})
                .get("application/json", {})
                .get("schema")
            )
            if req_body:
                schema["properties"]["requestBody"] = req_body

            params = spec.get("parameters", [])
            if params:
                param_properties = {
                    param["name"]: param["schema"]
                    for param in params
                    if "schema" in param
                }
                schema["properties"]["parameters"] = {
                    "type": "object",
                    "properties": param_properties,
                }

            functions.append(
                {"type": "function", "function": {"name": function_name, "description": desc, "parameters": schema}}
            )

    return functions


key: _node_content
string_value:
{"id_": "b74e8ffc-3e08-4c4e-8a7b-b2485c568ec4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "062d4c9c-61fe-46dd-9a8e-2c7af4618d6e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c1f5fae0f60c235c4fdc99cfd7a95f74a5b918480c7c4f5a938ca69b416804e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0fa8fc55-bdcf-4000-af95-5567bf238769", "node_type": "1", "metadata": {}, "hash": "848cc58d07658b84bf4550ea8edc6f7d900e23f4f064798e1e2b4042c7f7cedf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9508, "end_char_idx": 14285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def openapi_to_functions(openapi_spec):
    functions = []

    for path, methods in openapi_spec["paths"].items():
        for method, spec_with_ref in methods.items():
            # 1. Resolve JSON references.
            spec = jsonref.replace_refs(spec_with_ref)

            # 2. Extract a name for the functions.
            function_name = spec.get("operationId")

            # 3. Extract a description and parameters.
            desc = spec.get("description") or spec.get("summary", "")

            schema = {"type": "object", "properties": {}}

            req_body = (
                spec.get("requestBody", {})
                .get("content", {})
                .get("application/json", {})
                .get("schema")
            )
            if req_body:
                schema["properties"]["requestBody"] = req_body

            params = spec.get("parameters", [])
            if params:
                param_properties = {
                    param["name"]: param["schema"]
                    for param in params
                    if "schema" in param
                }
                schema["properties"]["parameters"] = {
                    "type": "object",
                    "properties": param_properties,
                }

            functions.append(
                {"type": "function", "function": {"name": function_name, "description": desc, "parameters": schema}}
            )

    return functions


functions = openapi_to_functions(openapi_spec)

for function in functions:
    pp(function)
    print()

```

    {'type': 'function',
     'function': {'name': 'listEvents',
                  'description': 'List all events',
                  'parameters': {'type': 'object', 'properties': {}}}}
    
    {'type': 'function',
     'function': {'name': 'createEvent',
                  'description': 'Create a new event',
                  'parameters': {'type': 'object',
                                 'properties': {'requestBody': {'type': 'object',
                                                                'properties': {'id': {'type': 'string'},
                                                                               'name': {'type': 'string'},
                                                                               'date': {'type': 'string',
                                                                                        'format': 'date-time'},
                                                                               'location': {'type': 'string'}},
                                                                'required': ['name',
                                                                             'date',
                                                                             'location']}}}}}
    
    {'type': 'function',
     'function': {'name': 'getEventById',
                  'description': 'Retrieve an event by ID',
                  'parameters': {'type': 'object',
                                 'properties': {'parameters': {'type': 'object',
                                                               'properties': {'id': {'type': 'string'}}}}}}}
    
    {'type': 'function',
     'function': {'name': 'deleteEvent',
                  'description': 'Delete an event by ID',
                  'parameters': {'type': 'object',
                                 'properties': {'parameters': {'type': 'object',
                                                               'properties': {'id': {'type': 'string'}}}}}}}
    
    {'type': 'function',
     'function': {'name': 'updateEventDetails',
                  'description': "Update an event's details by ID",
                  'parameters': {'type': 'object',
                                 'properties': {'requestBody': {'type': 'object',
                                                                'properties': {'name': {'type': 'string'},
                                                                               'date': {'type': 'string',
                                                                                        'format': 'date-time'},
                                                                               'location': {'type': 'string'}},
                                                                'required': ['name',
                                                                             'date',
                                                                             'location']},
                                                'parameters': {'type': 'object',
                                                               'properties': {'id': {'type': 'string'}}}}}}}
    


## How to call these functions with GPT


key: _node_content
string_value:
{"id_": "0fa8fc55-bdcf-4000-af95-5567bf238769", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b74e8ffc-3e08-4c4e-8a7b-b2485c568ec4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c92e78ca4f5f886f0d6dc9d7aa26cc2686d4333b279e2177f3f9bb7b23ebbd70", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a88ec60e-f480-465e-b4e7-88dcb180e47d", "node_type": "1", "metadata": {}, "hash": "8c489a9d682f644d4b336b35d49b13a5b3efd7e843252d84ec0bb95187e468ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14246, "end_char_idx": 16488, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## How to call these functions with GPT


Now that we have these function definitions, we can leverage GPT to call them intelligently based on user inputs.

It's important to note that the chat completions API does not execute the function; instead, it generates the JSON that you can use to call the function in your own code.

For more information on function-calling, refer to our dedicated [function-calling guide](./How_to_call_functions_with_chat_models.ipynb).



```python
SYSTEM_MESSAGE = """
You are a helpful assistant.
Respond to the following prompt by using function_call and then summarize actions.
Ask for clarification if a user request is ambiguous.
"""

# Maximum number of function calls allowed to prevent infinite or lengthy loops
MAX_CALLS = 5


def get_openai_response(functions, messages):
    return client.chat.completions.create(
        model="gpt-3.5-turbo-16k",
        tools=functions,
        tool_choice="auto",  # "auto" means the model can pick between generating a message or calling a function.
        temperature=0,
        messages=messages,
    )


def process_user_instruction(functions, instruction):
    num_calls = 0
    messages = [
        {"content": SYSTEM_MESSAGE, "role": "system"},
        {"content": instruction, "role": "user"},
    ]

    while num_calls < MAX_CALLS:
        response = get_openai_response(functions, messages)
        message = response.choices[0].message
        print(message)
        try:
            print(f"\n>> Function call #: {num_calls + 1}\n")
            pp(message.tool_calls)
            messages.append(message)

            # For the sake of this example, we'll simply add a message to simulate success.
            # Normally, you'd want to call the function here, and append the results to messages.
            messages.append(
                {
                    "role": "tool",
                    "content": "success",
                    "tool_call_id": message.tool_calls[0].id,
                }
            )

            num_calls += 1
        except:
            print("\n>> Message:\n")
            print(message.content)
            break

    if num_calls >= MAX_CALLS:
        print(f"Reached max chained function calls: {MAX_CALLS}")


key: _node_content
string_value:
{"id_": "a88ec60e-f480-465e-b4e7-88dcb180e47d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0fa8fc55-bdcf-4000-af95-5567bf238769", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "6a4e7baeb0b753d7731dbe063476ab3fb75ade0eb41c4de611e76f0f7807c69c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "948b739b-0887-44cc-b28f-a60e883975f1", "node_type": "1", "metadata": {}, "hash": "89289eab3468bfa2033e07e37caeab357cb7b632ce0dfe1a781d2fe6dbea0296", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14717, "end_char_idx": 18865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
SYSTEM_MESSAGE = """
You are a helpful assistant.
Respond to the following prompt by using function_call and then summarize actions.
Ask for clarification if a user request is ambiguous.
"""

# Maximum number of function calls allowed to prevent infinite or lengthy loops
MAX_CALLS = 5


def get_openai_response(functions, messages):
    return client.chat.completions.create(
        model="gpt-3.5-turbo-16k",
        tools=functions,
        tool_choice="auto",  # "auto" means the model can pick between generating a message or calling a function.
        temperature=0,
        messages=messages,
    )


def process_user_instruction(functions, instruction):
    num_calls = 0
    messages = [
        {"content": SYSTEM_MESSAGE, "role": "system"},
        {"content": instruction, "role": "user"},
    ]

    while num_calls < MAX_CALLS:
        response = get_openai_response(functions, messages)
        message = response.choices[0].message
        print(message)
        try:
            print(f"\n>> Function call #: {num_calls + 1}\n")
            pp(message.tool_calls)
            messages.append(message)

            # For the sake of this example, we'll simply add a message to simulate success.
            # Normally, you'd want to call the function here, and append the results to messages.
            messages.append(
                {
                    "role": "tool",
                    "content": "success",
                    "tool_call_id": message.tool_calls[0].id,
                }
            )

            num_calls += 1
        except:
            print("\n>> Message:\n")
            print(message.content)
            break

    if num_calls >= MAX_CALLS:
        print(f"Reached max chained function calls: {MAX_CALLS}")


USER_INSTRUCTION = """
Instruction: Get all the events.
Then create a new event named AGI Party.
Then delete event with id 2456.
"""

process_user_instruction(functions, USER_INSTRUCTION)

```

    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_jmlvEyMRMvOtB80adX9RbqIV', function=Function(arguments='{}', name='listEvents'), type='function')])
    
    >> Function call #: 1
    
    [ChatCompletionMessageToolCall(id='call_jmlvEyMRMvOtB80adX9RbqIV', function=Function(arguments='{}', name='listEvents'), type='function')]
    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_OOPOY7IHMq3T7Ib71JozlUQJ', function=Function(arguments='{\n  "requestBody": {\n    "id": "1234",\n    "name": "AGI Party",\n    "date": "2022-12-31",\n    "location": "New York"\n  }\n}', name='createEvent'), type='function')])
    
    >> Function call #: 2
    
    [ChatCompletionMessageToolCall(id='call_OOPOY7IHMq3T7Ib71JozlUQJ', function=Function(arguments='{\n  "requestBody": {\n    "id": "1234",\n    "name": "AGI Party",\n    "date": "2022-12-31",\n    "location": "New York"\n  }\n}', name='createEvent'), type='function')]
    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_Kxluu3fJSOsZNNCn3JIlWAAM', function=Function(arguments='{\n  "parameters": {\n    "id": "2456"\n  }\n}', name='deleteEvent'), type='function')])
    
    >> Function call #: 3
    
    [ChatCompletionMessageToolCall(id='call_Kxluu3fJSOsZNNCn3JIlWAAM', function=Function(arguments='{\n  "parameters": {\n    "id": "2456"\n  }\n}', name='deleteEvent'), type='function')]
    ChatCompletionMessage(content='Here are the actions I performed:\n\n1. Retrieved all the events.\n2. Created a new event named "AGI Party" with the ID "1234", scheduled for December 31, 2022, in New York.\n3. Deleted the event with the ID "2456".', role='assistant', function_call=None, tool_calls=None)
    
    >> Function call #: 4
    
    None
    
    >> Message:
    
    Here are the actions I performed:
    
    1. Retrieved all the events.
    2. Created a new event named "AGI Party" with the ID "1234", scheduled for December 31, 2022, in New York.
    3. Deleted the event with the ID "2456".


key: _node_content
string_value:
{"id_": "948b739b-0887-44cc-b28f-a60e883975f1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "17b1b5b7-ad6d-45ef-9291-5c677bf8a092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d9cedaceedb772db16d4de6d0966b9609ad9ea13fe095ad66a98dd8adaf2a5e3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a88ec60e-f480-465e-b4e7-88dcb180e47d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Function_calling_with_an_OpenAPI_spec.ipynb", "title": "Function-calling with an OpenAPI specification", "tokens": 4377, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "53bd9ab31d953adb7606a39f2604ffc7c77ec966c9d58533a16274b6842f6c55", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18868, "end_char_idx": 19410, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Conclusion

We have demonstrated how to convert OpenAPI specs into function specifications that can be given to GPT for it to intelligently call them, and shown how these can be chained together to perform complex operations.

Possible extensions of this system could include handling more complex user instructions that require conditional logic or looping, integrating with real APIs to perform actual operations, and improving error handling and validation to ensure the instructions are feasible and the function calls are successful.


key: _node_content
string_value:
{"id_": "4a6d9cf0-595f-4c21-9efd-e08f4cdaf4af", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3bc75bb7-dd7d-4788-805a-7b39b6c5eab1", "node_type": "1", "metadata": {}, "hash": "f19327d32a838ac329cc673258b0bd4b9f61853e1cb407f151dc3a9154e14c9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to stream completions

By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.

If you're generating long completions, waiting for the response can take many seconds.

To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.

To stream completions, set `stream=True` when calling the chat completions or completions endpoints. This will return an object that streams back the response as [data-only server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). Extract chunks from the `delta` field rather than the `message` field.

## Downsides

Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. This may have implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).

## Example code

Below, this notebook shows:
1. What a typical chat completion response looks like
2. What a streaming chat completion response looks like
3. How much time is saved by streaming a chat completion
4. How to get token usage data for streamed chat completion response


```python
# !pip install openai
```


```python
# imports
import time  # for measuring time duration of API calls
from openai import OpenAI
import os
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. What a typical chat completion response looks like

With a typical ChatCompletions API call, the response is first computed and then returned all at once.


key: _node_content
string_value:
{"id_": "3bc75bb7-dd7d-4788-805a-7b39b6c5eab1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a6d9cf0-595f-4c21-9efd-e08f4cdaf4af", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "8dd351c7a97c390f7420f561ca36dfaaa4fd1f5c8875d2e5a3ca26ae4660aa72", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "812ff326-fc6e-41e0-ab03-f6dd08770fae", "node_type": "1", "metadata": {}, "hash": "005efff13db7f0180872f8e879cc3d54467c9984564ac5ee8e217521a0e3d4ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to stream completions

By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.

If you're generating long completions, waiting for the response can take many seconds.

To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.

To stream completions, set `stream=True` when calling the chat completions or completions endpoints. This will return an object that streams back the response as [data-only server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). Extract chunks from the `delta` field rather than the `message` field.

## Downsides

Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. This may have implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).

## Example code

Below, this notebook shows:
1. What a typical chat completion response looks like
2. What a streaming chat completion response looks like
3. How much time is saved by streaming a chat completion
4. How to get token usage data for streamed chat completion response


```python
# !pip install openai
```


```python
# imports
import time  # for measuring time duration of API calls
from openai import OpenAI
import os
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. What a typical chat completion response looks like

With a typical ChatCompletions API call, the response is first computed and then returned all at once.


```python
# Example of an OpenAI ChatCompletion request
# https://platform.openai.com/docs/guides/text-generation/chat-completions-api

# record the time before the request is sent
start_time = time.time()

# send a ChatCompletion request to count to 100
response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[
        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}
    ],
    temperature=0,
)
# calculate the time it took to receive the response
response_time = time.time() - start_time

# print the time delay and text received
print(f"Full response received {response_time:.2f} seconds after request")
print(f"Full response received:\n{response}")

```

    Full response received 1.88 seconds after request
    Full response received:
    ChatCompletion(id='chatcmpl-9lMgdoiMfxVHPDNVCtvXuTWcQ2GGb', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100', role='assistant', function_call=None, tool_calls=None))], created=1721075651, model='gpt-july-test', object='chat.completion', system_fingerprint='fp_e9b8ed65d2', usage=CompletionUsage(completion_tokens=298, prompt_tokens=36, total_tokens=334))


key: _node_content
string_value:
{"id_": "812ff326-fc6e-41e0-ab03-f6dd08770fae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3bc75bb7-dd7d-4788-805a-7b39b6c5eab1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "79b3f2b880a759c14e0a6f4f61cf1f57cbd1385ac616ed79073ddfe7c9cef1f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6aedbbce-1ab9-4c3d-908b-0164331dd917", "node_type": "1", "metadata": {}, "hash": "ac916f2d332902c67eb053fd839078736f445348dd1e5a82cb4aaad43eed1e4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3480, "end_char_idx": 4744, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The reply can be extracted with `response.choices[0].message`.

The content of the reply can be extracted with `response.choices[0].message.content`.


```python
reply = response.choices[0].message
print(f"Extracted reply: \n{reply}")

reply_content = response.choices[0].message.content
print(f"Extracted content: \n{reply_content}")

```

    Extracted reply: 
    ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100', role='assistant', function_call=None, tool_calls=None)
    Extracted content: 
    1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100


key: _node_content
string_value:
{"id_": "6aedbbce-1ab9-4c3d-908b-0164331dd917", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "812ff326-fc6e-41e0-ab03-f6dd08770fae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ff82b6f6e5404d684130d000bd1b2122c7e9f852ae7ad48e6019c4a80da0c7c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf5726ee-12f4-4264-a83b-9350bddcc37a", "node_type": "1", "metadata": {}, "hash": "ef6f9c1af415f37e91b90b1e154b5ada9453c801582d4aa752af14f769138842", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4747, "end_char_idx": 7494, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 2. How to stream a chat completion

With a streaming API call, the response is sent back incrementally in chunks via an [event stream](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). In Python, you can iterate over these events with a `for` loop.

Let's see what it looks like:


```python
# Example of an OpenAI ChatCompletion request with stream=True
# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream

# a ChatCompletion request
response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[
        {'role': 'user', 'content': "What's 1+1? Answer in one word."}
    ],
    temperature=0,
    stream=True  # this time, we set stream=True
)

for chunk in response:
    print(chunk)
    print(chunk.choices[0].delta.content)
    print("****************")
```

    ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)
    
    ****************
    ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='Two', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)
    Two
    ****************
    ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content='.', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)
    .
    ****************
    ChatCompletionChunk(id='chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)], created=1721075653, model='gpt-july-test', object='chat.completion.chunk', system_fingerprint='fp_e9b8ed65d2', usage=None)
    None
    ****************


As you can see above, streaming responses have a `delta` field rather than a `message` field. `delta` can hold things like:
- a role token (e.g., `{"role": "assistant"}`)
- a content token (e.g., `{"content": "\n\n"}`)
- nothing (e.g., `{}`), when the stream is over

### 3. How much time is saved by streaming a chat completion

Now let's ask `gpt-4o-mini` to count to 100 again, and see how long it takes.


key: _node_content
string_value:
{"id_": "cf5726ee-12f4-4264-a83b-9350bddcc37a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6aedbbce-1ab9-4c3d-908b-0164331dd917", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a1546d955dce714e8e39259ca3ae15cdb83589e431909b7800e5036186721430", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9624315-27ee-43b3-b433-08303e309adb", "node_type": "1", "metadata": {}, "hash": "779bc69aeff0fdd1ed72b851d81a2391ccd79794595fcc30778b32c2a8521097", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As you can see above, streaming responses have a `delta` field rather than a `message` field. `delta` can hold things like:
- a role token (e.g., `{"role": "assistant"}`)
- a content token (e.g., `{"content": "\n\n"}`)
- nothing (e.g., `{}`), when the stream is over

### 3. How much time is saved by streaming a chat completion

Now let's ask `gpt-4o-mini` to count to 100 again, and see how long it takes.


```python
# Example of an OpenAI ChatCompletion request with stream=True
# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream

# record the time before the request is sent
start_time = time.time()

# send a ChatCompletion request to count to 100
response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[
        {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g. 1, 2, 3, .'}
    ],
    temperature=0,
    stream=True  # again, we set stream=True
)
# create variables to collect the stream of chunks
collected_chunks = []
collected_messages = []
# iterate through the stream of events
for chunk in response:
    chunk_time = time.time() - start_time  # calculate the time delay of the chunk
    collected_chunks.append(chunk)  # save the event response
    chunk_message = chunk.choices[0].delta.content  # extract the message
    collected_messages.append(chunk_message)  # save the message
    print(f"Message received {chunk_time:.2f} seconds after request: {chunk_message}")  # print the delay and text

# print the time delay and text received
print(f"Full response received {chunk_time:.2f} seconds after request")
# clean None in collected_messages
collected_messages = [m for m in collected_messages if m is not None]
full_reply_content = ''.join(collected_messages)
print(f"Full conversation received: {full_reply_content}")

```

    Message received 1.14 seconds after request: 
    Message received 1.14 seconds after request: 1
    Message received 1.14 seconds after request: ,
    Message received 1.14 seconds after request:  
    Message received 1.14 seconds after request: 2
    Message received 1.16 seconds after request: ,
    Message received 1.16 seconds after request:  
    Message received 1.16 seconds after request: 3
    Message received 1.35 seconds after request: ,
    Message received 1.35 seconds after request:  
    Message received 1.35 seconds after request: 4
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 5
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 6
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 7
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.


key: _node_content
string_value:
{"id_": "e9624315-27ee-43b3-b433-08303e309adb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf5726ee-12f4-4264-a83b-9350bddcc37a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "5e755082de09a2f71dd7c1d3a55f81b91506f0bed130f09e58eb354270122fcd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a08bcf9-1463-40c4-a5dd-c3bac12753c6", "node_type": "1", "metadata": {}, "hash": "e933aaa3447735fdea2b308e50d5c1fe195176a3537232f889e8a4a1e2295d82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8531, "end_char_idx": 11619, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2f} seconds after request: {chunk_message}")  # print the delay and text

# print the time delay and text received
print(f"Full response received {chunk_time:.2f} seconds after request")
# clean None in collected_messages
collected_messages = [m for m in collected_messages if m is not None]
full_reply_content = ''.join(collected_messages)
print(f"Full conversation received: {full_reply_content}")

```

    Message received 1.14 seconds after request: 
    Message received 1.14 seconds after request: 1
    Message received 1.14 seconds after request: ,
    Message received 1.14 seconds after request:  
    Message received 1.14 seconds after request: 2
    Message received 1.16 seconds after request: ,
    Message received 1.16 seconds after request:  
    Message received 1.16 seconds after request: 3
    Message received 1.35 seconds after request: ,
    Message received 1.35 seconds after request:  
    Message received 1.35 seconds after request: 4
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 5
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 6
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 7
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 8
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 9
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 10
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 11
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 12
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.45 seconds after request: 13
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.45 seconds after request: 14
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.45 seconds after request: 15
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.46 seconds after request: 16
    Message received 1.46 seconds after request: ,
    Message received 1.46 seconds after request:  
    Message received 1.47 seconds after request: 17
    Message received 1.47 seconds after request: ,
    Message received 1.47 seconds after request:  
    Message received 1.


key: _node_content
string_value:
{"id_": "3a08bcf9-1463-40c4-a5dd-c3bac12753c6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9624315-27ee-43b3-b433-08303e309adb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "749878ff42746ec3990c13ab9c70d650317c812a0344828dd290c90fddbbbffe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b351b993-9d63-4108-bd27-509f5c826a44", "node_type": "1", "metadata": {}, "hash": "f0fc35973dc7c9ea1a318144fb7fa672bb9a80d659ab774b140d9cabad8de82c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10081, "end_char_idx": 13108, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
36 seconds after request: 8
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 9
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 10
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 11
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.36 seconds after request: 12
    Message received 1.36 seconds after request: ,
    Message received 1.36 seconds after request:  
    Message received 1.45 seconds after request: 13
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.45 seconds after request: 14
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.45 seconds after request: 15
    Message received 1.45 seconds after request: ,
    Message received 1.45 seconds after request:  
    Message received 1.46 seconds after request: 16
    Message received 1.46 seconds after request: ,
    Message received 1.46 seconds after request:  
    Message received 1.47 seconds after request: 17
    Message received 1.47 seconds after request: ,
    Message received 1.47 seconds after request:  
    Message received 1.49 seconds after request: 18
    Message received 1.49 seconds after request: ,
    Message received 1.49 seconds after request:  
    Message received 1.52 seconds after request: 19
    Message received 1.52 seconds after request: ,
    Message received 1.52 seconds after request:  
    Message received 1.53 seconds after request: 20
    Message received 1.53 seconds after request: ,
    Message received 1.53 seconds after request:  
    Message received 1.55 seconds after request: 21
    Message received 1.55 seconds after request: ,
    Message received 1.55 seconds after request:  
    Message received 1.56 seconds after request: 22
    Message received 1.56 seconds after request: ,
    Message received 1.56 seconds after request:  
    Message received 1.58 seconds after request: 23
    Message received 1.58 seconds after request: ,
    Message received 1.58 seconds after request:  
    Message received 1.59 seconds after request: 24
    Message received 1.59 seconds after request: ,
    Message received 1.59 seconds after request:  
    Message received 1.62 seconds after request: 25
    Message received 1.62 seconds after request: ,
    Message received 1.62 seconds after request:  
    Message received 1.62 seconds after request: 26
    Message received 1.62 seconds after request: ,
    Message received 1.62 seconds after request:  
    Message received 1.65 seconds after request: 27
    Message received 1.65 seconds after request: ,
    Message received 1.


key: _node_content
string_value:
{"id_": "b351b993-9d63-4108-bd27-509f5c826a44", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a08bcf9-1463-40c4-a5dd-c3bac12753c6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bd418ed7d1f91a6337cc005f63443e7567be11c13b819fa08723832fb361383a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c9c193a-e44e-4b63-8762-b01716209cb4", "node_type": "1", "metadata": {}, "hash": "b7af32aeb1fa9366341e3d56383a84ce543ab2fa36fc9625e753f2aceb2538d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11549, "end_char_idx": 14597, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Message received 1.47 seconds after request:  
    Message received 1.49 seconds after request: 18
    Message received 1.49 seconds after request: ,
    Message received 1.49 seconds after request:  
    Message received 1.52 seconds after request: 19
    Message received 1.52 seconds after request: ,
    Message received 1.52 seconds after request:  
    Message received 1.53 seconds after request: 20
    Message received 1.53 seconds after request: ,
    Message received 1.53 seconds after request:  
    Message received 1.55 seconds after request: 21
    Message received 1.55 seconds after request: ,
    Message received 1.55 seconds after request:  
    Message received 1.56 seconds after request: 22
    Message received 1.56 seconds after request: ,
    Message received 1.56 seconds after request:  
    Message received 1.58 seconds after request: 23
    Message received 1.58 seconds after request: ,
    Message received 1.58 seconds after request:  
    Message received 1.59 seconds after request: 24
    Message received 1.59 seconds after request: ,
    Message received 1.59 seconds after request:  
    Message received 1.62 seconds after request: 25
    Message received 1.62 seconds after request: ,
    Message received 1.62 seconds after request:  
    Message received 1.62 seconds after request: 26
    Message received 1.62 seconds after request: ,
    Message received 1.62 seconds after request:  
    Message received 1.65 seconds after request: 27
    Message received 1.65 seconds after request: ,
    Message received 1.65 seconds after request:  
    Message received 1.67 seconds after request: 28
    Message received 1.67 seconds after request: ,
    Message received 1.67 seconds after request:  
    Message received 1.69 seconds after request: 29
    Message received 1.69 seconds after request: ,
    Message received 1.69 seconds after request:  
    Message received 1.80 seconds after request: 30
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 31
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 32
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 33
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 34
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 35
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 36
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.82 seconds after request: 37
    Message received 1.


key: _node_content
string_value:
{"id_": "0c9c193a-e44e-4b63-8762-b01716209cb4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b351b993-9d63-4108-bd27-509f5c826a44", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "3dbeb3e2f19887c67eb495442c8bc20418933e3fb9a76f6263e0d8ff7f500bf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c99104c-72e9-4c82-a3b0-5d7545e39019", "node_type": "1", "metadata": {}, "hash": "01d3ed79cd8f6611ace1320414834ddecc40eea4210524827fc05ef963c989e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13057, "end_char_idx": 16085, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
65 seconds after request: ,
    Message received 1.65 seconds after request:  
    Message received 1.67 seconds after request: 28
    Message received 1.67 seconds after request: ,
    Message received 1.67 seconds after request:  
    Message received 1.69 seconds after request: 29
    Message received 1.69 seconds after request: ,
    Message received 1.69 seconds after request:  
    Message received 1.80 seconds after request: 30
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 31
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 32
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 33
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 34
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 35
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.80 seconds after request: 36
    Message received 1.80 seconds after request: ,
    Message received 1.80 seconds after request:  
    Message received 1.82 seconds after request: 37
    Message received 1.82 seconds after request: ,
    Message received 1.82 seconds after request:  
    Message received 1.83 seconds after request: 38
    Message received 1.83 seconds after request: ,
    Message received 1.83 seconds after request:  
    Message received 1.84 seconds after request: 39
    Message received 1.84 seconds after request: ,
    Message received 1.84 seconds after request:  
    Message received 1.87 seconds after request: 40
    Message received 1.87 seconds after request: ,
    Message received 1.87 seconds after request:  
    Message received 1.88 seconds after request: 41
    Message received 1.88 seconds after request: ,
    Message received 1.88 seconds after request:  
    Message received 1.91 seconds after request: 42
    Message received 1.91 seconds after request: ,
    Message received 1.91 seconds after request:  
    Message received 1.93 seconds after request: 43
    Message received 1.93 seconds after request: ,
    Message received 1.93 seconds after request:  
    Message received 1.93 seconds after request: 44
    Message received 1.93 seconds after request: ,
    Message received 1.93 seconds after request:  
    Message received 1.95 seconds after request: 45
    Message received 1.95 seconds after request: ,
    Message received 1.95 seconds after request:  
    Message received 2.00 seconds after request: 46
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.


key: _node_content
string_value:
{"id_": "1c99104c-72e9-4c82-a3b0-5d7545e39019", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c9c193a-e44e-4b63-8762-b01716209cb4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "3f255cca9a7f77f8cfb8a701d8a6a2bbcfabba98e056f8f6a940f55d1df7445b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ab2a2d8-df76-47e5-8221-62bb7e56b28e", "node_type": "1", "metadata": {}, "hash": "e373b8bb67795f0e105c05d36666c8abab17a55233444120239d2153e0956e03", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14545, "end_char_idx": 17574, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
82 seconds after request: 37
    Message received 1.82 seconds after request: ,
    Message received 1.82 seconds after request:  
    Message received 1.83 seconds after request: 38
    Message received 1.83 seconds after request: ,
    Message received 1.83 seconds after request:  
    Message received 1.84 seconds after request: 39
    Message received 1.84 seconds after request: ,
    Message received 1.84 seconds after request:  
    Message received 1.87 seconds after request: 40
    Message received 1.87 seconds after request: ,
    Message received 1.87 seconds after request:  
    Message received 1.88 seconds after request: 41
    Message received 1.88 seconds after request: ,
    Message received 1.88 seconds after request:  
    Message received 1.91 seconds after request: 42
    Message received 1.91 seconds after request: ,
    Message received 1.91 seconds after request:  
    Message received 1.93 seconds after request: 43
    Message received 1.93 seconds after request: ,
    Message received 1.93 seconds after request:  
    Message received 1.93 seconds after request: 44
    Message received 1.93 seconds after request: ,
    Message received 1.93 seconds after request:  
    Message received 1.95 seconds after request: 45
    Message received 1.95 seconds after request: ,
    Message received 1.95 seconds after request:  
    Message received 2.00 seconds after request: 46
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 47
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 48
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 49
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 50
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 51
    Message received 2.00 seconds after request: ,
    Message received 2.04 seconds after request:  
    Message received 2.04 seconds after request: 52
    Message received 2.04 seconds after request: ,
    Message received 2.04 seconds after request:  
    Message received 2.04 seconds after request: 53
    Message received 2.04 seconds after request: ,
    Message received 2.13 seconds after request:  
    Message received 2.13 seconds after request: 54
    Message received 2.14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.14 seconds after request: 55
    Message received 2.14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.14 seconds after request: 56
    Message received 2.14 seconds after request: ,
    Message received 2.


key: _node_content
string_value:
{"id_": "4ab2a2d8-df76-47e5-8221-62bb7e56b28e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c99104c-72e9-4c82-a3b0-5d7545e39019", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0e084c296705d592ffdb1a7d7ff87bd98bf72e6ed02509c8e967ba9e2a689fdf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c88893da-db43-485a-b0d7-739fc04c8078", "node_type": "1", "metadata": {}, "hash": "5d1551b475dd6cd0b5e3075c8b3260f5a1edfc05bee9458dfd05c39a041e25c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16015, "end_char_idx": 19063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 47
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 48
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 49
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 50
    Message received 2.00 seconds after request: ,
    Message received 2.00 seconds after request:  
    Message received 2.00 seconds after request: 51
    Message received 2.00 seconds after request: ,
    Message received 2.04 seconds after request:  
    Message received 2.04 seconds after request: 52
    Message received 2.04 seconds after request: ,
    Message received 2.04 seconds after request:  
    Message received 2.04 seconds after request: 53
    Message received 2.04 seconds after request: ,
    Message received 2.13 seconds after request:  
    Message received 2.13 seconds after request: 54
    Message received 2.14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.14 seconds after request: 55
    Message received 2.14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.14 seconds after request: 56
    Message received 2.14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.16 seconds after request: 57
    Message received 2.16 seconds after request: ,
    Message received 2.16 seconds after request:  
    Message received 2.17 seconds after request: 58
    Message received 2.17 seconds after request: ,
    Message received 2.17 seconds after request:  
    Message received 2.19 seconds after request: 59
    Message received 2.19 seconds after request: ,
    Message received 2.19 seconds after request:  
    Message received 2.21 seconds after request: 60
    Message received 2.21 seconds after request: ,
    Message received 2.21 seconds after request:  
    Message received 2.34 seconds after request: 61
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 62
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 63
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 64
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 65
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 66
    Message received 2.


key: _node_content
string_value:
{"id_": "c88893da-db43-485a-b0d7-739fc04c8078", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ab2a2d8-df76-47e5-8221-62bb7e56b28e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "e6555779e7f71fd072de2d19f5306da8c0d5a3e5873645d358a046ac62babe32", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42f67c89-1d94-49ee-a3d1-79886f30ee35", "node_type": "1", "metadata": {}, "hash": "28a893365ef4430fa3782d06243d3d0f9bf42f202812d020076ad8c02d5ef537", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17523, "end_char_idx": 20551, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
14 seconds after request: ,
    Message received 2.14 seconds after request:  
    Message received 2.16 seconds after request: 57
    Message received 2.16 seconds after request: ,
    Message received 2.16 seconds after request:  
    Message received 2.17 seconds after request: 58
    Message received 2.17 seconds after request: ,
    Message received 2.17 seconds after request:  
    Message received 2.19 seconds after request: 59
    Message received 2.19 seconds after request: ,
    Message received 2.19 seconds after request:  
    Message received 2.21 seconds after request: 60
    Message received 2.21 seconds after request: ,
    Message received 2.21 seconds after request:  
    Message received 2.34 seconds after request: 61
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 62
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 63
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 64
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 65
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 66
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 67
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.36 seconds after request: 68
    Message received 2.36 seconds after request: ,
    Message received 2.36 seconds after request:  
    Message received 2.36 seconds after request: 69
    Message received 2.36 seconds after request: ,
    Message received 2.36 seconds after request:  
    Message received 2.38 seconds after request: 70
    Message received 2.38 seconds after request: ,
    Message received 2.38 seconds after request:  
    Message received 2.39 seconds after request: 71
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 72
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 73
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 74
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 75
    Message received 2.39 seconds after request: ,
    Message received 2.40 seconds after request:  
    Message received 2.


key: _node_content
string_value:
{"id_": "42f67c89-1d94-49ee-a3d1-79886f30ee35", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c88893da-db43-485a-b0d7-739fc04c8078", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "8a9524853dcd9451e7d557b10818f5a466ce408b7c3ae19a5c768b9916101e74", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b056edea-e5f0-4fd7-8406-847b66c6ce8c", "node_type": "1", "metadata": {}, "hash": "b636dfe2181dfdfeb0dbdd009acec1f21ac55c3f7718ac3892b0e4cc6137d3b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19011, "end_char_idx": 22040, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
34 seconds after request: 66
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.34 seconds after request: 67
    Message received 2.34 seconds after request: ,
    Message received 2.34 seconds after request:  
    Message received 2.36 seconds after request: 68
    Message received 2.36 seconds after request: ,
    Message received 2.36 seconds after request:  
    Message received 2.36 seconds after request: 69
    Message received 2.36 seconds after request: ,
    Message received 2.36 seconds after request:  
    Message received 2.38 seconds after request: 70
    Message received 2.38 seconds after request: ,
    Message received 2.38 seconds after request:  
    Message received 2.39 seconds after request: 71
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 72
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 73
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 74
    Message received 2.39 seconds after request: ,
    Message received 2.39 seconds after request:  
    Message received 2.39 seconds after request: 75
    Message received 2.39 seconds after request: ,
    Message received 2.40 seconds after request:  
    Message received 2.40 seconds after request: 76
    Message received 2.40 seconds after request: ,
    Message received 2.42 seconds after request:  
    Message received 2.42 seconds after request: 77
    Message received 2.42 seconds after request: ,
    Message received 2.51 seconds after request:  
    Message received 2.51 seconds after request: 78
    Message received 2.51 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 79
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 80
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 81
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 82
    Message received 2.52 seconds after request: ,
    Message received 2.60 seconds after request:  
    Message received 2.60 seconds after request: 83
    Message received 2.60 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.64 seconds after request: 84
    Message received 2.64 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.64 seconds after request: 85
    Message received 2.64 seconds after request: ,
    Message received 2.


key: _node_content
string_value:
{"id_": "b056edea-e5f0-4fd7-8406-847b66c6ce8c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "42f67c89-1d94-49ee-a3d1-79886f30ee35", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a989135aeba4face88b4e88d3e816052e28c5fa59dfdcfb232941f72ada0d799", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6551eaa-ea15-4e4d-b9b9-ed16212e023a", "node_type": "1", "metadata": {}, "hash": "b59df920697612fb222dc49f40cd05ff0f7b18da5e334c60757c57815ee3cb22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20481, "end_char_idx": 23529, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Message received 2.40 seconds after request:  
    Message received 2.40 seconds after request: 76
    Message received 2.40 seconds after request: ,
    Message received 2.42 seconds after request:  
    Message received 2.42 seconds after request: 77
    Message received 2.42 seconds after request: ,
    Message received 2.51 seconds after request:  
    Message received 2.51 seconds after request: 78
    Message received 2.51 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 79
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 80
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 81
    Message received 2.52 seconds after request: ,
    Message received 2.52 seconds after request:  
    Message received 2.52 seconds after request: 82
    Message received 2.52 seconds after request: ,
    Message received 2.60 seconds after request:  
    Message received 2.60 seconds after request: 83
    Message received 2.60 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.64 seconds after request: 84
    Message received 2.64 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.64 seconds after request: 85
    Message received 2.64 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.66 seconds after request: 86
    Message received 2.66 seconds after request: ,
    Message received 2.66 seconds after request:  
    Message received 2.66 seconds after request: 87
    Message received 2.66 seconds after request: ,
    Message received 2.66 seconds after request:  
    Message received 2.68 seconds after request: 88
    Message received 2.68 seconds after request: ,
    Message received 2.68 seconds after request:  
    Message received 2.69 seconds after request: 89
    Message received 2.69 seconds after request: ,
    Message received 2.69 seconds after request:  
    Message received 2.72 seconds after request: 90
    Message received 2.72 seconds after request: ,
    Message received 2.72 seconds after request:  
    Message received 2.82 seconds after request: 91
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 92
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 93
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 94
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 95
    Message received 2.


key: _node_content
string_value:
{"id_": "b6551eaa-ea15-4e4d-b9b9-ed16212e023a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b056edea-e5f0-4fd7-8406-847b66c6ce8c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d2ec27dd53ad670a2c23f3db08e89062880db1f48bfcdaf9c704e7b2b9285db3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f66fad73-155a-4bb3-bf46-04f8a35af49e", "node_type": "1", "metadata": {}, "hash": "4f939bed0b7e1404426396724df577d12c16802e12332af75cb6a4a63f66eb29", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21989, "end_char_idx": 24627, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
64 seconds after request: ,
    Message received 2.64 seconds after request:  
    Message received 2.66 seconds after request: 86
    Message received 2.66 seconds after request: ,
    Message received 2.66 seconds after request:  
    Message received 2.66 seconds after request: 87
    Message received 2.66 seconds after request: ,
    Message received 2.66 seconds after request:  
    Message received 2.68 seconds after request: 88
    Message received 2.68 seconds after request: ,
    Message received 2.68 seconds after request:  
    Message received 2.69 seconds after request: 89
    Message received 2.69 seconds after request: ,
    Message received 2.69 seconds after request:  
    Message received 2.72 seconds after request: 90
    Message received 2.72 seconds after request: ,
    Message received 2.72 seconds after request:  
    Message received 2.82 seconds after request: 91
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 92
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 93
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 94
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 95
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 96
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 97
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 98
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 99
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 100
    Message received 2.82 seconds after request: None
    Full response received 2.82 seconds after request
    Full conversation received: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,


key: _node_content
string_value:
{"id_": "f66fad73-155a-4bb3-bf46-04f8a35af49e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6551eaa-ea15-4e4d-b9b9-ed16212e023a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "36ac56e9b39402acdcd12d705d93dc75a30624ff2abcd86c82ed478586fbc290", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9b343fb-31f6-41ad-8097-5e7656fcd7ae", "node_type": "1", "metadata": {}, "hash": "80c3ef75ea4a379416a1aedbf3a57accd6de5353e65855ffc42c1de7208a8c74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23477, "end_char_idx": 25846, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
82 seconds after request: 95
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 96
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 97
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 98
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 99
    Message received 2.82 seconds after request: ,
    Message received 2.82 seconds after request:  
    Message received 2.82 seconds after request: 100
    Message received 2.82 seconds after request: None
    Full response received 2.82 seconds after request
    Full conversation received: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100


#### Time comparison

In the example above, both requests took about 4 to 5 seconds to fully complete. Request times will vary depending on load and other stochastic factors.

However, with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds.

### 4. How to get token usage data for streamed chat completion response

You can get token usage statistics for your streamed response by setting `stream_options={"include_usage": True}`. When you do so, an extra chunk will be streamed as the final chunk. You can access the usage data for the entire request via the `usage` field on this chunk. A few important notes when you set `stream_options={"include_usage": True}`:
* The value for the `usage` field on all chunks except for the last one will be null.
* The `usage` field on the last chunk contains token usage statistics for the entire request.
* The `choices` field on the last chunk will always be an empty array `[]`.

Let's see how it works using the example in 2.


key: _node_content
string_value:
{"id_": "b9b343fb-31f6-41ad-8097-5e7656fcd7ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3b7c77bc-a243-4713-890e-4afae30ee21e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b617d332020ce29478029df8701ec2601f023e5fed5b3d394be06e7b9cc4bc57", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f66fad73-155a-4bb3-bf46-04f8a35af49e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_stream_completions.ipynb", "title": "How to stream completions", "tokens": 7537, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c4596fa6133a2419ae74d290692c765a6e85ec55ff6fde72c08d05b9be697db4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24584, "end_char_idx": 27263, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100


#### Time comparison

In the example above, both requests took about 4 to 5 seconds to fully complete. Request times will vary depending on load and other stochastic factors.

However, with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds.

### 4. How to get token usage data for streamed chat completion response

You can get token usage statistics for your streamed response by setting `stream_options={"include_usage": True}`. When you do so, an extra chunk will be streamed as the final chunk. You can access the usage data for the entire request via the `usage` field on this chunk. A few important notes when you set `stream_options={"include_usage": True}`:
* The value for the `usage` field on all chunks except for the last one will be null.
* The `usage` field on the last chunk contains token usage statistics for the entire request.
* The `choices` field on the last chunk will always be an empty array `[]`.

Let's see how it works using the example in 2.


```python
# Example of an OpenAI ChatCompletion request with stream=True and stream_options={"include_usage": True}

# a ChatCompletion request
response = client.chat.completions.create(
    model='gpt-4o-mini',
    messages=[
        {'role': 'user', 'content': "What's 1+1? Answer in one word."}
    ],
    temperature=0,
    stream=True,
    stream_options={"include_usage": True}, # retrieving token usage for stream response
)

for chunk in response:
    print(f"choices: {chunk.choices}\nusage: {chunk.usage}")
    print("****************")
```

    choices: [Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)]
    usage: None
    ****************
    choices: [Choice(delta=ChoiceDelta(content='Two', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)]
    usage: None
    ****************
    choices: [Choice(delta=ChoiceDelta(content='.', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)]
    usage: None
    ****************
    choices: [Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)]
    usage: None
    ****************
    choices: []
    usage: CompletionUsage(completion_tokens=2, prompt_tokens=18, total_tokens=20)
    ****************


key: _node_content
string_value:
{"id_": "c1996a2a-0ecd-44dd-a0d6-d1c1aa8a22bd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5064503-5b4a-43a9-81ce-b35ed3d86727", "node_type": "1", "metadata": {}, "hash": "3ad3c2ce5822375dcc0da17307d60a2b6a9cc4717930c72646785072c48f8108", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3539, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using logprobs for classification and Q&A evaluation

This notebook demonstrates the use of the `logprobs` parameter in the Chat Completions API. When `logprobs` is enabled, the API returns the log probabilities of each output token, along with a limited number of the most likely tokens at each token position and their log probabilities. The relevant request parameters are:
* `logprobs`: Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. This option is currently not available on the `gpt-4-vision-preview` model.
* `top_logprobs`: An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used.

Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about `logprobs`:
* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered.
* Logprob can be any negative number or `0.0`. `0.0` corresponds to 100% probability.
* Logprobs allow us to compute the joint probability of a sequence as the sum of the logprobs of the individual tokens. This is useful for scoring and ranking model outputs. Another common approach is to take the average per-token logprob of a sentence to choose the best generation.
* We can examine the `logprobs` assigned to different candidate tokens to understand what options the model considered plausible or implausible.

While there are a wide array of use cases for `logprobs`, this notebook will focus on its use for:

1. Classification tasks

* Large Language Models excel at many classification tasks, but accurately measuring the model's confidence in its outputs can be challenging. `logprobs` provide a probability associated with each class prediction, enabling users to set their own classification or confidence thresholds.

2. Retrieval (Q&A) evaluation

* `logprobs` can assist with self-evaluation in retrieval applications. In the Q&A example, the model outputs a contrived `has_sufficient_context_for_answer` boolean, which can serve as a confidence score of whether the answer is contained in the retrieved content. Evaluations of this type can reduce retrieval-based hallucinations and enhance accuracy.

3. Autocomplete
*  `logprobs` could help us decide how to suggest words as a user is typing.

4. Token highlighting and outputting bytes
*  Users can easily create a token highlighter using the built in tokenization that comes with enabling `logprobs`. Additionally, the bytes parameter includes the ASCII encoding of each output character, which is particularly useful for reproducing emojis and special characters.

5. Calculating perplexity
* `logprobs` can be used to help us assess the model's overall confidence in a result and help us compare the confidence of results from different prompts.

## 0. Imports and utils


```python
from openai import OpenAI
from math import exp
import numpy as np
from IPython.display import display, HTML
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```


key: _node_content
string_value:
{"id_": "e5064503-5b4a-43a9-81ce-b35ed3d86727", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1996a2a-0ecd-44dd-a0d6-d1c1aa8a22bd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "963b9a365b6a761184c1f5d0c46949f402bd7a78ff1b2ed794d6209dd8161249", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c8b8832-c1ca-4349-821c-f6ed53d4c132", "node_type": "1", "metadata": {}, "hash": "024c34cf60ed7f8e2380259c61ac4e300d58c468a2b72b463ea1dea66c2680ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3302, "end_char_idx": 6133, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from openai import OpenAI
from math import exp
import numpy as np
from IPython.display import display, HTML
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```


```python
def get_completion(
    messages: list[dict[str, str]],
    model: str = "gpt-4",
    max_tokens=500,
    temperature=0,
    stop=None,
    seed=123,
    tools=None,
    logprobs=None,  # whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message..
    top_logprobs=None,
) -> str:
    params = {
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": temperature,
        "stop": stop,
        "seed": seed,
        "logprobs": logprobs,
        "top_logprobs": top_logprobs,
    }
    if tools:
        params["tools"] = tools

    completion = client.chat.completions.create(**params)
    return completion
```

## 1. Using `logprobs` to assess confidence for classification tasks

Let's say we want to create a system to classify news articles into a set of pre-defined categories. Without `logprobs`, we can use Chat Completions to do this, but it is much more difficult to assess the certainty with which the model made its classifications.

Now, with `logprobs` enabled, we can see exactly how confident the model is in its predictions, which is crucial for creating an accurate and trustworthy classifier. For example, if the log probability for the chosen category is high, this suggests the model is quite confident in its classification. If it's low, this suggests the model is less confident. This can be particularly useful in cases where the model's classification is not what you expected, or when the model's output needs to be reviewed or validated by a human.

We'll begin with a prompt that presents the model with four categories: **Technology, Politics, Sports, and Arts**. The model is then tasked with classifying articles into these categories based solely on their headlines.


```python
CLASSIFICATION_PROMPT = """You will be given a headline of a news article.
Classify the article into one of the following categories: Technology, Politics, Sports, and Art.
Return only the name of the category, and nothing else.
MAKE SURE your output is one of the four categories stated.
Article headline: {headline}"""

```

Let's look at three sample headlines, and first begin with a standard Chat Completions output, without `logprobs`


```python
headlines = [
    "Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.",
    "Local Mayor Launches Initiative to Enhance Urban Public Transport.",
    "Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut",
]

```


key: _node_content
string_value:
{"id_": "9c8b8832-c1ca-4349-821c-f6ed53d4c132", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5064503-5b4a-43a9-81ce-b35ed3d86727", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "207c09768243b69e734d7beb641467a957bae480a9d90f2fdeb3419de8f66ae4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95f4203d-6d2a-463d-af0d-c5fb4b0cb9f6", "node_type": "1", "metadata": {}, "hash": "3b1b84bcabcb5a68c81422d9abc9862f320d16cb80b9873c5c2fead61c394c99", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5412, "end_char_idx": 8163, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
CLASSIFICATION_PROMPT = """You will be given a headline of a news article.
Classify the article into one of the following categories: Technology, Politics, Sports, and Art.
Return only the name of the category, and nothing else.
MAKE SURE your output is one of the four categories stated.
Article headline: {headline}"""

```

Let's look at three sample headlines, and first begin with a standard Chat Completions output, without `logprobs`


```python
headlines = [
    "Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.",
    "Local Mayor Launches Initiative to Enhance Urban Public Transport.",
    "Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut",
]

```


```python
for headline in headlines:
    print(f"\nHeadline: {headline}")
    API_RESPONSE = get_completion(
        [{"role": "user", "content": CLASSIFICATION_PROMPT.format(headline=headline)}],
        model="gpt-4",
    )
    print(f"Category: {API_RESPONSE.choices[0].message.content}\n")
```

    
    Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.
    Category: Technology
    
    
    Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.
    Category: Politics
    
    
    Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut
    Category: Art
    


Here we can see the selected category for each headline. However, we have no visibility into the confidence of the model in its predictions. Let's rerun the same prompt but with `logprobs` enabled, and `top_logprobs` set to 2 (this will show us the 2 most likely output tokens for each token). Additionally we can also output the linear probability of each output token, in order to convert the log probability to the more easily interprable scale of 0-100%. 


```python
for headline in headlines:
    print(f"\nHeadline: {headline}")
    API_RESPONSE = get_completion(
        [{"role": "user", "content": CLASSIFICATION_PROMPT.format(headline=headline)}],
        model="gpt-4",
        logprobs=True,
        top_logprobs=2,
    )
    top_two_logprobs = API_RESPONSE.choices[0].logprobs.content[0].top_logprobs
    html_content = ""
    for i, logprob in enumerate(top_two_logprobs, start=1):
        html_content += (
            f"<span style='color: cyan'>Output token {i}:</span> {logprob.token}, "
            f"<span style='color: darkorange'>logprobs:</span> {logprob.logprob}, "
            f"<span style='color: magenta'>linear probability:</span> {np.round(np.exp(logprob.logprob)*100,2)}%<br>"
        )
    display(HTML(html_content))
    print("\n")
```

    
    Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.


key: _node_content
string_value:
{"id_": "95f4203d-6d2a-463d-af0d-c5fb4b0cb9f6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c8b8832-c1ca-4349-821c-f6ed53d4c132", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "71ee9129058e40d1680fa8aa9dcfdc74c8be343cda439fcfe71ea95622aa5ef7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b023c0e-0ef9-409b-9c41-cb57cb5f629f", "node_type": "1", "metadata": {}, "hash": "179497ab64bc9f56d95e4bad86d779293e6b4a9eda260c441d23579e3bcabfc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6792, "end_char_idx": 9487, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here we can see the selected category for each headline. However, we have no visibility into the confidence of the model in its predictions. Let's rerun the same prompt but with `logprobs` enabled, and `top_logprobs` set to 2 (this will show us the 2 most likely output tokens for each token). Additionally we can also output the linear probability of each output token, in order to convert the log probability to the more easily interprable scale of 0-100%. 


```python
for headline in headlines:
    print(f"\nHeadline: {headline}")
    API_RESPONSE = get_completion(
        [{"role": "user", "content": CLASSIFICATION_PROMPT.format(headline=headline)}],
        model="gpt-4",
        logprobs=True,
        top_logprobs=2,
    )
    top_two_logprobs = API_RESPONSE.choices[0].logprobs.content[0].top_logprobs
    html_content = ""
    for i, logprob in enumerate(top_two_logprobs, start=1):
        html_content += (
            f"<span style='color: cyan'>Output token {i}:</span> {logprob.token}, "
            f"<span style='color: darkorange'>logprobs:</span> {logprob.logprob}, "
            f"<span style='color: magenta'>linear probability:</span> {np.round(np.exp(logprob.logprob)*100,2)}%<br>"
        )
    display(HTML(html_content))
    print("\n")
```

    
    Headline: Tech Giant Unveils Latest Smartphone Model with Advanced Photo-Editing Features.



<span style='color: cyan'>Output token 1:</span> Technology, <span style='color: darkorange'>logprobs:</span> -2.4584822e-06, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> Techn, <span style='color: darkorange'>logprobs:</span> -13.781253, <span style='color: magenta'>linear probability:</span> 0.0%<br>


    
    
    
    Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.



<span style='color: cyan'>Output token 1:</span> Politics, <span style='color: darkorange'>logprobs:</span> -2.4584822e-06, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> Technology, <span style='color: darkorange'>logprobs:</span> -13.937503, <span style='color: magenta'>linear probability:</span> 0.0%<br>


    
    
    
    Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut



<span style='color: cyan'>Output token 1:</span> Art, <span style='color: darkorange'>logprobs:</span> -0.009169078, <span style='color: magenta'>linear probability:</span> 99.09%<br><span style='color: cyan'>Output token 2:</span> Sports, <span style='color: darkorange'>logprobs:</span> -4.696669, <span style='color: magenta'>linear probability:</span> 0.91%<br>


key: _node_content
string_value:
{"id_": "0b023c0e-0ef9-409b-9c41-cb57cb5f629f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95f4203d-6d2a-463d-af0d-c5fb4b0cb9f6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "24076ef590de35e4c4115576ea4797eea0ae64450d44f84d494254cb728a10b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0788c8e-37bf-48df-bc9f-124bb1a23510", "node_type": "1", "metadata": {}, "hash": "f7b70d541e04fe42394d57c33feb604e7ed832b0d6448dcdd75c8e85ef828ad9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8562, "end_char_idx": 10722, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Headline: Local Mayor Launches Initiative to Enhance Urban Public Transport.



<span style='color: cyan'>Output token 1:</span> Politics, <span style='color: darkorange'>logprobs:</span> -2.4584822e-06, <span style='color: magenta'>linear probability:</span> 100.0%<br><span style='color: cyan'>Output token 2:</span> Technology, <span style='color: darkorange'>logprobs:</span> -13.937503, <span style='color: magenta'>linear probability:</span> 0.0%<br>


    
    
    
    Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut



<span style='color: cyan'>Output token 1:</span> Art, <span style='color: darkorange'>logprobs:</span> -0.009169078, <span style='color: magenta'>linear probability:</span> 99.09%<br><span style='color: cyan'>Output token 2:</span> Sports, <span style='color: darkorange'>logprobs:</span> -4.696669, <span style='color: magenta'>linear probability:</span> 0.91%<br>


    
    


As expected from the first two headlines, `gpt-4` is nearly 100% confident in its classifications, as the content is clearly technology and politics focused respectively. However, the third headline combines both sports and art-related themes, so we see the model is less confident in its selection.

This shows how important using `logprobs` can be, as if we are using LLMs for classification tasks we can set confidence theshholds, or output several potential output tokens if the log probability of the selected output is not sufficiently high. For instance, if we are creating a recommendation engine to tag articles, we can automatically classify headlines crossing a certain threshold, and send the less certain headlines for manual review.

## 2. Retrieval confidence scoring to reduce hallucinations

To reduce hallucinations, and the performance of our RAG-based Q&A system, we can use `logprobs` to evaluate how confident the model is in its retrieval.

Let's say we have built a retrieval system using RAG for Q&A, but are struggling with hallucinated answers to our questions. *Note:* we will use a hardcoded article for this example, but see other entries in the cookbook for tutorials on using RAG for Q&A.


