# embedding_metadata

key: _node_content
string_value:
{"id_": "b59f247d-59c6-46af-9d0a-d0c248f0a3d6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05e80a33-7263-44d8-9d06-be5a4cedeff5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "86376ea4c35da130a9719e19aa61682e93f636b63b217d1f097216eaa55e29e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5709a4cc-73d4-4a04-985a-12996391f50e", "node_type": "1", "metadata": {}, "hash": "06220c279e32d2014db649da58b80d9cd665c8f447c7e0a607405d607633b444", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14209, "end_char_idx": 16513, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# An example of a system message that primes the assistant to give brief, to-the-point answers
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a laconic assistant. You reply with brief, to-the-point answers with no elaboration."},
        {"role": "user", "content": "Can you explain how fractions work?"},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    Fractions represent parts of a whole. They have a numerator (top number) and a denominator (bottom number).


### Few-shot prompting

In some cases, it's easier to show the model what you want rather than tell the model what you want.

One way to show the model what you want is with faked example messages.

For example:


```python
# An example of a faked few-shot conversation to prime the model into translating business jargon to simpler speech
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful, pattern-following assistant."},
        {"role": "user", "content": "Help me translate the following corporate jargon into plain English."},
        {"role": "assistant", "content": "Sure, I'd be happy to!"},
        {"role": "user", "content": "New synergies will help drive top-line growth."},
        {"role": "assistant", "content": "Things working well together will increase revenue."},
        {"role": "user", "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage."},
        {"role": "assistant", "content": "Let's talk later when we're less busy about how to do better."},
        {"role": "user", "content": "This late pivot means we don't have time to boil the ocean for the client deliverable."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    This sudden change in direction means we don't have enough time to complete the entire project for the client.


To help clarify that the example messages are not part of a real conversation, and shouldn't be referred back to by the model, you can try setting the `name` field of `system` messages to `example_user` and `example_assistant`.

Transforming the few-shot example above, we could write:


key: _node_content
string_value:
{"id_": "5709a4cc-73d4-4a04-985a-12996391f50e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b59f247d-59c6-46af-9d0a-d0c248f0a3d6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "08493cd93b54b482fcf5463b7c7918d1ff89c7bfb9a2261c880eadd2e964b87f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc6258fd-6a91-40c0-b669-325a4587734d", "node_type": "1", "metadata": {}, "hash": "a0f65e9c971a3294da9c5dd81b8f9ae2d3757e66a1eb36cd849b8d1b7a5bc9ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14792, "end_char_idx": 17687, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Few-shot prompting

In some cases, it's easier to show the model what you want rather than tell the model what you want.

One way to show the model what you want is with faked example messages.

For example:


```python
# An example of a faked few-shot conversation to prime the model into translating business jargon to simpler speech
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful, pattern-following assistant."},
        {"role": "user", "content": "Help me translate the following corporate jargon into plain English."},
        {"role": "assistant", "content": "Sure, I'd be happy to!"},
        {"role": "user", "content": "New synergies will help drive top-line growth."},
        {"role": "assistant", "content": "Things working well together will increase revenue."},
        {"role": "user", "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage."},
        {"role": "assistant", "content": "Let's talk later when we're less busy about how to do better."},
        {"role": "user", "content": "This late pivot means we don't have time to boil the ocean for the client deliverable."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    This sudden change in direction means we don't have enough time to complete the entire project for the client.


To help clarify that the example messages are not part of a real conversation, and shouldn't be referred back to by the model, you can try setting the `name` field of `system` messages to `example_user` and `example_assistant`.

Transforming the few-shot example above, we could write:


```python
# The business jargon translation example, but with example names for the example messages
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English."},
        {"role": "system", "name":"example_user", "content": "New synergies will help drive top-line growth."},
        {"role": "system", "name": "example_assistant", "content": "Things working well together will increase revenue."},
        {"role": "system", "name":"example_user", "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage."},
        {"role": "system", "name": "example_assistant", "content": "Let's talk later when we're less busy about how to do better."},
        {"role": "user", "content": "This late pivot means we don't have time to boil the ocean for the client deliverable."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    This sudden change in direction means we don't have enough time to complete the entire project for the client.


key: _node_content
string_value:
{"id_": "fc6258fd-6a91-40c0-b669-325a4587734d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5709a4cc-73d4-4a04-985a-12996391f50e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b14506571f331ceb27c15d3b80a4982f8db34d9506c9cc8ef24e615c1f8a4799", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0822f0d6-0819-4f64-93ce-90d58938c9e3", "node_type": "1", "metadata": {}, "hash": "754ee44774e55da89def1b74200167b08ab434651b0f63797a2c78529b2c4cc4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16228, "end_char_idx": 19231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To help clarify that the example messages are not part of a real conversation, and shouldn't be referred back to by the model, you can try setting the `name` field of `system` messages to `example_user` and `example_assistant`.

Transforming the few-shot example above, we could write:


```python
# The business jargon translation example, but with example names for the example messages
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English."},
        {"role": "system", "name":"example_user", "content": "New synergies will help drive top-line growth."},
        {"role": "system", "name": "example_assistant", "content": "Things working well together will increase revenue."},
        {"role": "system", "name":"example_user", "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage."},
        {"role": "system", "name": "example_assistant", "content": "Let's talk later when we're less busy about how to do better."},
        {"role": "user", "content": "This late pivot means we don't have time to boil the ocean for the client deliverable."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    This sudden change in direction means we don't have enough time to complete the entire project for the client.


Not every attempt at engineering conversations will succeed at first.

If your first attempts fail, don't be afraid to experiment with different ways of priming or conditioning the model.

As an example, one developer discovered an increase in accuracy when they inserted a user message that said "Great job so far, these have been perfect" to help condition the model into providing higher quality responses.

For more ideas on how to lift the reliability of the models, consider reading our guide on [techniques to increase reliability](../techniques_to_improve_reliability). It was written for non-chat models, but many of its principles still apply.

## 4. Counting tokens

When you submit your request, the API transforms the messages into a sequence of tokens.

The number of tokens used affects:
- the cost of the request
- the time it takes to generate the response
- when the reply gets cut off from hitting the maximum token limit (4,096 for `gpt-3.5-turbo` or 8,192 for `gpt-4`)

You can use the following function to count the number of tokens that a list of messages will use.

Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee. 

In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.

Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).


```python
import tiktoken


key: _node_content
string_value:
{"id_": "0822f0d6-0819-4f64-93ce-90d58938c9e3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc6258fd-6a91-40c0-b669-325a4587734d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a253c0e6309d68729a8ce94287f63abd1424b918ac278913349884c1d9acbad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e51a0f6-7e8c-4ff8-84e0-c2a10b6484c0", "node_type": "1", "metadata": {}, "hash": "164d8dac14ebb7ce276d08e7721b8418829202f24e76840094a511d7a7a7027d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17690, "end_char_idx": 20960, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Not every attempt at engineering conversations will succeed at first.

If your first attempts fail, don't be afraid to experiment with different ways of priming or conditioning the model.

As an example, one developer discovered an increase in accuracy when they inserted a user message that said "Great job so far, these have been perfect" to help condition the model into providing higher quality responses.

For more ideas on how to lift the reliability of the models, consider reading our guide on [techniques to increase reliability](../techniques_to_improve_reliability). It was written for non-chat models, but many of its principles still apply.

## 4. Counting tokens

When you submit your request, the API transforms the messages into a sequence of tokens.

The number of tokens used affects:
- the cost of the request
- the time it takes to generate the response
- when the reply gets cut off from hitting the maximum token limit (4,096 for `gpt-3.5-turbo` or 8,192 for `gpt-4`)

You can use the following function to count the number of tokens that a list of messages will use.

Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee. 

In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.

Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).


```python
import tiktoken


def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows <|start|>{role/name}\n{content}<|end|>\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens

```


key: _node_content
string_value:
{"id_": "7e51a0f6-7e8c-4ff8-84e0-c2a10b6484c0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0822f0d6-0819-4f64-93ce-90d58938c9e3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f840160a4ac4e4c522506fff8f3293db6cb9b7d8a28e8de0c8a5d6fec9d42749", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20963, "end_char_idx": 23594, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# let's verify the function above matches the OpenAI API response
example_messages = [
    {
        "role": "system",
        "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "New synergies will help drive top-line growth.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Things working well together will increase revenue.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Let's talk later when we're less busy about how to do better.",
    },
    {
        "role": "user",
        "content": "This late pivot means we don't have time to boil the ocean for the client deliverable.",
    },
]

for model in [
    # "gpt-3.5-turbo-0301",
    # "gpt-4-0314",
    # "gpt-4-0613",
    "gpt-3.5-turbo-1106",
    "gpt-3.5-turbo",
    "gpt-4",
    "gpt-4-1106-preview",
    ]:
    print(model)
    # example token count from the function defined above
    print(f"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().")
    # example token count from the OpenAI API
    response = client.chat.completions.create(model=model,
    messages=example_messages,
    temperature=0,
    max_tokens=1)
    token = response.usage.prompt_tokens
    print(f'{token} prompt tokens counted by the OpenAI API.')
    print()

```

    gpt-3.5-turbo-1106
    Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-3.5-turbo
    Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-4
    Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-4-1106-preview
    Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb


key: _node_content
string_value:
{"id_": "0ba96b33-938d-49c9-b26f-5c09f063aa73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c64b6f3-66a8-4917-bcf8-3c59bb68862b", "node_type": "1", "metadata": {}, "hash": "38402decf5664123218753b91568d62c3e7c4d7de554935f21d6d68fadf56395", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2582, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to call functions with chat models

This notebook covers how to use the Chat Completions API in combination with external functions to extend the capabilities of GPT models.

`tools` is an optional parameter in the Chat Completion API which can be used to provide function specifications. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. Note that the API will not actually execute any function calls. It is up to developers to execute function calls using model outputs.

Within the `tools` parameter, if the `functions` parameter is provided then by default the model will decide when it is appropriate to use one of the functions. The API can be forced to use a specific function by setting the `tool_choice` parameter to `{"type": "function", "function": {"name": "my_function"}}`. The API can also be forced to not use any function by setting the `tool_choice` parameter to `"none"`. If a function is used, the output will contain `"finish_reason": "tool_calls"` in the response, as well as a `tool_calls` object that has the name of the function and the generated function arguments.

### Overview

This notebook contains the following 2 sections:

- **How to generate function arguments:** Specify a set of functions and use the API to generate function arguments.
- **How to call functions with model generated arguments:** Close the loop by actually executing functions with model generated arguments.

## How to generate function arguments


```python
!pip install scipy --quiet
!pip install tenacity --quiet
!pip install tiktoken --quiet
!pip install termcolor --quiet
!pip install openai --quiet
```


```python
import json
from openai import OpenAI
from tenacity import retry, wait_random_exponential, stop_after_attempt
from termcolor import colored  

GPT_MODEL = "gpt-4o"
client = OpenAI()
```

### Utilities

First let's define a few utilities for making calls to the Chat Completions API and for maintaining and keeping track of the conversation state.


```python
@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, tools=None, tool_choice=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=tools,
            tool_choice=tool_choice,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


key: _node_content
string_value:
{"id_": "1c64b6f3-66a8-4917-bcf8-3c59bb68862b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ba96b33-938d-49c9-b26f-5c09f063aa73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a076e03d2d9e2c4d42adb7502ed6cf0a9d2275c3ed702fe1960cef953fccc594", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb063d73-d5bb-4237-befa-2b735954249f", "node_type": "1", "metadata": {}, "hash": "c25252aca9500956ac40183d2226d83c21cdbebf8d8bfb6719d5760c9ac56e3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1525, "end_char_idx": 3849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install scipy --quiet
!pip install tenacity --quiet
!pip install tiktoken --quiet
!pip install termcolor --quiet
!pip install openai --quiet
```


```python
import json
from openai import OpenAI
from tenacity import retry, wait_random_exponential, stop_after_attempt
from termcolor import colored  

GPT_MODEL = "gpt-4o"
client = OpenAI()
```

### Utilities

First let's define a few utilities for making calls to the Chat Completions API and for maintaining and keeping track of the conversation state.


```python
@retry(wait=wait_random_exponential(multiplier=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, tools=None, tool_choice=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=tools,
            tool_choice=tool_choice,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


```python
def pretty_print_conversation(messages):
    role_to_color = {
        "system": "red",
        "user": "green",
        "assistant": "blue",
        "function": "magenta",
    }
    
    for message in messages:
        if message["role"] == "system":
            print(colored(f"system: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "user":
            print(colored(f"user: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "assistant" and message.get("function_call"):
            print(colored(f"assistant: {message['function_call']}\n", role_to_color[message["role"]]))
        elif message["role"] == "assistant" and not message.get("function_call"):
            print(colored(f"assistant: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "function":
            print(colored(f"function ({message['name']}): {message['content']}\n", role_to_color[message["role"]]))

```

### Basic concepts

Let's create some function specifications to interface with a hypothetical weather API. We'll pass these function specification to the Chat Completions API in order to generate function arguments that adhere to the specification.


key: _node_content
string_value:
{"id_": "bb063d73-d5bb-4237-befa-2b735954249f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c64b6f3-66a8-4917-bcf8-3c59bb68862b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "abd69efb8bb5c93a2e30c42c907db9ef1f6963ae8c74a3737b67befb0c2ccb6b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "14097e7d-b134-43f4-b139-d3f72acd526c", "node_type": "1", "metadata": {}, "hash": "022db31fedbfbff67071453c29d490ced17f7f6fa8b6be2e235c1103efc36968", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2585, "end_char_idx": 6219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def pretty_print_conversation(messages):
    role_to_color = {
        "system": "red",
        "user": "green",
        "assistant": "blue",
        "function": "magenta",
    }
    
    for message in messages:
        if message["role"] == "system":
            print(colored(f"system: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "user":
            print(colored(f"user: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "assistant" and message.get("function_call"):
            print(colored(f"assistant: {message['function_call']}\n", role_to_color[message["role"]]))
        elif message["role"] == "assistant" and not message.get("function_call"):
            print(colored(f"assistant: {message['content']}\n", role_to_color[message["role"]]))
        elif message["role"] == "function":
            print(colored(f"function ({message['name']}): {message['content']}\n", role_to_color[message["role"]]))

```

### Basic concepts

Let's create some function specifications to interface with a hypothetical weather API. We'll pass these function specification to the Chat Completions API in order to generate function arguments that adhere to the specification.


```python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                },
                "required": ["location", "format"],
            },
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_n_day_weather_forecast",
            "description": "Get an N-day weather forecast",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                    "format": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit to use. Infer this from the users location.",
                    },
                    "num_days": {
                        "type": "integer",
                        "description": "The number of days to forecast",
                    }
                },
                "required": ["location", "format", "num_days"]
            },
        }
    },
]
```

If we prompt the model about the current weather, it will respond with some clarifying questions.


```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "What's the weather like today"})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
assistant_message

```


key: _node_content
string_value:
{"id_": "14097e7d-b134-43f4-b139-d3f72acd526c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb063d73-d5bb-4237-befa-2b735954249f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "32bd698ecf01536b785d19429ec428ce38239ba8580dbfd4d5546a46a2ff45ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ef95b666-c1e7-4a1b-8d9d-ed7ebe3ae2b2", "node_type": "1", "metadata": {}, "hash": "172f20db166ba3a65f270dc2f6e81aabb188299f42e8761f2b1f28d440d12a53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5769, "end_char_idx": 8918, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "What's the weather like today"})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
assistant_message

```




    ChatCompletionMessage(content='Sure, can you please provide me with the name of your city and state?', role='assistant', function_call=None, tool_calls=None)



Once we provide the missing information, it will generate the appropriate function arguments for us.


```python
messages.append({"role": "user", "content": "I'm in Glasgow, Scotland."})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
assistant_message

```




    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_xb7QwwNnx90LkmhtlW0YrgP2', function=Function(arguments='{"location":"Glasgow, Scotland","format":"celsius"}', name='get_current_weather'), type='function')])



By prompting it differently, we can get it to target the other function we've told it about.


```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "what is the weather going to be like in Glasgow, Scotland over the next x days"})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
assistant_message

```




    ChatCompletionMessage(content='To provide you with the weather forecast for Glasgow, Scotland, could you please specify the number of days you would like the forecast for?', role='assistant', function_call=None, tool_calls=None)



Once again, the model is asking us for clarification because it doesn't have enough information yet. In this case it already knows the location for the forecast, but it needs to know how many days are required in the forecast.


```python
messages.append({"role": "user", "content": "5 days"})
chat_response = chat_completion_request(
    messages, tools=tools
)
chat_response.choices[0]

```




    Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_34PBraFdNN6KR95uD5rHF8Aw', function=Function(arguments='{"location":"Glasgow, Scotland","format":"celsius","num_days":5}', name='get_n_day_weather_forecast'), type='function')]))



#### Forcing the use of specific functions or no function

We can force the model to use a specific function, for example get_n_day_weather_forecast by using the function_call argument. By doing so, we force the model to make assumptions about how to use it.


key: _node_content
string_value:
{"id_": "ef95b666-c1e7-4a1b-8d9d-ed7ebe3ae2b2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "14097e7d-b134-43f4-b139-d3f72acd526c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "09c61301f13c79f99355f041db7e5edab6275dd5be42217a98dbffb0d9114500", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e2910b8-c3a8-49cd-bf1d-c9fb073613e9", "node_type": "1", "metadata": {}, "hash": "c406b2b49109cc4139f8e07a224881a1a1f24da3eed54ca64fa76e89cef6838d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7145, "end_char_idx": 10248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "what is the weather going to be like in Glasgow, Scotland over the next x days"})
chat_response = chat_completion_request(
    messages, tools=tools
)
assistant_message = chat_response.choices[0].message
messages.append(assistant_message)
assistant_message

```




    ChatCompletionMessage(content='To provide you with the weather forecast for Glasgow, Scotland, could you please specify the number of days you would like the forecast for?', role='assistant', function_call=None, tool_calls=None)



Once again, the model is asking us for clarification because it doesn't have enough information yet. In this case it already knows the location for the forecast, but it needs to know how many days are required in the forecast.


```python
messages.append({"role": "user", "content": "5 days"})
chat_response = chat_completion_request(
    messages, tools=tools
)
chat_response.choices[0]

```




    Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_34PBraFdNN6KR95uD5rHF8Aw', function=Function(arguments='{"location":"Glasgow, Scotland","format":"celsius","num_days":5}', name='get_n_day_weather_forecast'), type='function')]))



#### Forcing the use of specific functions or no function

We can force the model to use a specific function, for example get_n_day_weather_forecast by using the function_call argument. By doing so, we force the model to make assumptions about how to use it.


```python
# in this cell we force the model to use get_n_day_weather_forecast
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me a weather report for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools, tool_choice={"type": "function", "function": {"name": "get_n_day_weather_forecast"}}
)
chat_response.choices[0].message
```




    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_FImGxrLowOAOszCaaQqQWmEN', function=Function(arguments='{"location":"Toronto, Canada","format":"celsius","num_days":7}', name='get_n_day_weather_forecast'), type='function')])




```python
# if we don't force the model to use get_n_day_weather_forecast it may not
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me a weather report for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools
)
chat_response.choices[0].message
```


key: _node_content
string_value:
{"id_": "5e2910b8-c3a8-49cd-bf1d-c9fb073613e9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ef95b666-c1e7-4a1b-8d9d-ed7ebe3ae2b2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dbf6eca1822502e1811d970be73cc7bbb8fd5f6bc2ac24a6964542249307eed1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b224e2cb-775b-4b68-bbb9-9b663e34fb75", "node_type": "1", "metadata": {}, "hash": "df9beb088672f6f75aafd9a9df37da403bae8eb37fb12158310eeb057b99eb07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8660, "end_char_idx": 11601, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Forcing the use of specific functions or no function

We can force the model to use a specific function, for example get_n_day_weather_forecast by using the function_call argument. By doing so, we force the model to make assumptions about how to use it.


```python
# in this cell we force the model to use get_n_day_weather_forecast
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me a weather report for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools, tool_choice={"type": "function", "function": {"name": "get_n_day_weather_forecast"}}
)
chat_response.choices[0].message
```




    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_FImGxrLowOAOszCaaQqQWmEN', function=Function(arguments='{"location":"Toronto, Canada","format":"celsius","num_days":7}', name='get_n_day_weather_forecast'), type='function')])




```python
# if we don't force the model to use get_n_day_weather_forecast it may not
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me a weather report for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools
)
chat_response.choices[0].message
```




    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_n84kYFqjNFDPNGDEnjnrd2KC', function=Function(arguments='{"location": "Toronto, Canada", "format": "celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_AEs3AFhJc9pn42hWSbHTaIDh', function=Function(arguments='{"location": "Toronto, Canada", "format": "celsius", "num_days": 3}', name='get_n_day_weather_forecast'), type='function')])



We can also force the model to not use a function at all. By doing so we prevent it from producing a proper function call.


```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me the current weather (use Celcius) for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools, tool_choice="none"
)
chat_response.choices[0].message

```




    ChatCompletionMessage(content="Sure, I'll get the current weather for Toronto, Canada in Celsius.", role='assistant', function_call=None, tool_calls=None)



### Parallel Function Calling

Newer models such as gpt-4o or gpt-3.5-turbo can call multiple functions in one turn.


key: _node_content
string_value:
{"id_": "b224e2cb-775b-4b68-bbb9-9b663e34fb75", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e2910b8-c3a8-49cd-bf1d-c9fb073613e9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3481b3fa94bd5ae7d393f985d88cb4b3a9fe70a74416a2d9911ca831127bdd1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ce01d4d-a180-4581-9109-c5203852f7ca", "node_type": "1", "metadata": {}, "hash": "8a11f50154f079af3ebee974f2265a1e7cb82a75d310223863c515701f312b9a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10257, "end_char_idx": 13337, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_n84kYFqjNFDPNGDEnjnrd2KC', function=Function(arguments='{"location": "Toronto, Canada", "format": "celsius"}', name='get_current_weather'), type='function'), ChatCompletionMessageToolCall(id='call_AEs3AFhJc9pn42hWSbHTaIDh', function=Function(arguments='{"location": "Toronto, Canada", "format": "celsius", "num_days": 3}', name='get_n_day_weather_forecast'), type='function')])



We can also force the model to not use a function at all. By doing so we prevent it from producing a proper function call.


```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "Give me the current weather (use Celcius) for Toronto, Canada."})
chat_response = chat_completion_request(
    messages, tools=tools, tool_choice="none"
)
chat_response.choices[0].message

```




    ChatCompletionMessage(content="Sure, I'll get the current weather for Toronto, Canada in Celsius.", role='assistant', function_call=None, tool_calls=None)



### Parallel Function Calling

Newer models such as gpt-4o or gpt-3.5-turbo can call multiple functions in one turn.


```python
messages = []
messages.append({"role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous."})
messages.append({"role": "user", "content": "what is the weather going to be like in San Francisco and Glasgow over the next 4 days"})
chat_response = chat_completion_request(
    messages, tools=tools, model=GPT_MODEL
)

assistant_message = chat_response.choices[0].message.tool_calls
assistant_message
```




    [ChatCompletionMessageToolCall(id='call_ObhLiJwaHwc3U1KyB4Pdpx8y', function=Function(arguments='{"location": "San Francisco, CA", "format": "fahrenheit", "num_days": 4}', name='get_n_day_weather_forecast'), type='function'),
     ChatCompletionMessageToolCall(id='call_5YRgeZ0MGBMFKE3hZiLouwg7', function=Function(arguments='{"location": "Glasgow, SCT", "format": "celsius", "num_days": 4}', name='get_n_day_weather_forecast'), type='function')]



## How to call functions with model generated arguments

In our next example, we'll demonstrate how to execute functions whose inputs are model-generated, and use this to implement an agent that can answer questions for us about a database. For simplicity we'll use the [Chinook sample database](https://www.sqlitetutorial.net/sqlite-sample-database/).

*Note:* SQL generation can be high-risk in a production environment since models are not perfectly reliable at generating correct SQL.

### Specifying a function to execute SQL queries

First let's define some helpful utility functions to extract data from a SQLite database.


```python
import sqlite3

conn = sqlite3.connect("data/Chinook.db")
print("Opened database successfully")
```

    Opened database successfully


key: _node_content
string_value:
{"id_": "4ce01d4d-a180-4581-9109-c5203852f7ca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b224e2cb-775b-4b68-bbb9-9b663e34fb75", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ad1e89b023af057defc29d96e30f0e6e7d4a591b3f064e836a748c463aaa8411", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c295668d-af35-4b34-badc-23f8e2e5af33", "node_type": "1", "metadata": {}, "hash": "40eea6c7cf635a5045b7cc0b459c1320f894f6f7bff0de859bccc766e22131ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12113, "end_char_idx": 14828, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[ChatCompletionMessageToolCall(id='call_ObhLiJwaHwc3U1KyB4Pdpx8y', function=Function(arguments='{"location": "San Francisco, CA", "format": "fahrenheit", "num_days": 4}', name='get_n_day_weather_forecast'), type='function'),
     ChatCompletionMessageToolCall(id='call_5YRgeZ0MGBMFKE3hZiLouwg7', function=Function(arguments='{"location": "Glasgow, SCT", "format": "celsius", "num_days": 4}', name='get_n_day_weather_forecast'), type='function')]



## How to call functions with model generated arguments

In our next example, we'll demonstrate how to execute functions whose inputs are model-generated, and use this to implement an agent that can answer questions for us about a database. For simplicity we'll use the [Chinook sample database](https://www.sqlitetutorial.net/sqlite-sample-database/).

*Note:* SQL generation can be high-risk in a production environment since models are not perfectly reliable at generating correct SQL.

### Specifying a function to execute SQL queries

First let's define some helpful utility functions to extract data from a SQLite database.


```python
import sqlite3

conn = sqlite3.connect("data/Chinook.db")
print("Opened database successfully")
```

    Opened database successfully



```python
def get_table_names(conn):
    """Return a list of table names."""
    table_names = []
    tables = conn.execute("SELECT name FROM sqlite_master WHERE type='table';")
    for table in tables.fetchall():
        table_names.append(table[0])
    return table_names


def get_column_names(conn, table_name):
    """Return a list of column names."""
    column_names = []
    columns = conn.execute(f"PRAGMA table_info('{table_name}');").fetchall()
    for col in columns:
        column_names.append(col[1])
    return column_names


def get_database_info(conn):
    """Return a list of dicts containing the table name and columns for each table in the database."""
    table_dicts = []
    for table_name in get_table_names(conn):
        columns_names = get_column_names(conn, table_name)
        table_dicts.append({"table_name": table_name, "column_names": columns_names})
    return table_dicts

```

Now can use these utility functions to extract a representation of the database schema.


```python
database_schema_dict = get_database_info(conn)
database_schema_string = "\n".join(
    [
        f"Table: {table['table_name']}\nColumns: {', '.join(table['column_names'])}"
        for table in database_schema_dict
    ]
)
```

As before, we'll define a function specification for the function we'd like the API to generate arguments for. Notice that we are inserting the database schema into the function specification. This will be important for the model to know about.


key: _node_content
string_value:
{"id_": "c295668d-af35-4b34-badc-23f8e2e5af33", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ce01d4d-a180-4581-9109-c5203852f7ca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "54d9a28a3f22bd90f63de3be7d56c3bfad7c8b75876f624c6d46e5961d222102", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58379fcd-b29d-465f-aaad-0299dcdfa07f", "node_type": "1", "metadata": {}, "hash": "713b5fbb7f0a80047521709fbd7b998d3c14e9ce32db67f0ad62d3fb004f4749", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13194, "end_char_idx": 16862, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import sqlite3

conn = sqlite3.connect("data/Chinook.db")
print("Opened database successfully")
```

    Opened database successfully



```python
def get_table_names(conn):
    """Return a list of table names."""
    table_names = []
    tables = conn.execute("SELECT name FROM sqlite_master WHERE type='table';")
    for table in tables.fetchall():
        table_names.append(table[0])
    return table_names


def get_column_names(conn, table_name):
    """Return a list of column names."""
    column_names = []
    columns = conn.execute(f"PRAGMA table_info('{table_name}');").fetchall()
    for col in columns:
        column_names.append(col[1])
    return column_names


def get_database_info(conn):
    """Return a list of dicts containing the table name and columns for each table in the database."""
    table_dicts = []
    for table_name in get_table_names(conn):
        columns_names = get_column_names(conn, table_name)
        table_dicts.append({"table_name": table_name, "column_names": columns_names})
    return table_dicts

```

Now can use these utility functions to extract a representation of the database schema.


```python
database_schema_dict = get_database_info(conn)
database_schema_string = "\n".join(
    [
        f"Table: {table['table_name']}\nColumns: {', '.join(table['column_names'])}"
        for table in database_schema_dict
    ]
)
```

As before, we'll define a function specification for the function we'd like the API to generate arguments for. Notice that we are inserting the database schema into the function specification. This will be important for the model to know about.


```python
tools = [
    {
        "type": "function",
        "function": {
            "name": "ask_database",
            "description": "Use this function to answer user questions about music. Input should be a fully formed SQL query.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": f"""
                                SQL query extracting info to answer the user's question.
                                SQL should be written using this database schema:
                                {database_schema_string}
                                The query should be returned in plain text, not in JSON.
                                """,
                    }
                },
                "required": ["query"],
            },
        }
    }
]
```

### Executing SQL queries

Now let's implement the function that will actually excute queries against the database.


```python
def ask_database(conn, query):
    """Function to query SQLite database with a provided SQL query."""
    try:
        results = str(conn.execute(query).fetchall())
    except Exception as e:
        results = f"query failed with error: {e}"
    return results
```

##### Steps to invoke a function call using Chat Completions API: 

**Step 1**: Prompt the model with content that may result in model selecting a tool to use. The description of the tools such as a function names and signature is defined in the 'Tools' list and passed to the model in API call. If selected, the function name and parameters are included in the response.<br>
  
**Step 2**: Check programmatically if model wanted to call a function. If true, proceed to step 3. <br>  
**Step 3**: Extract the function name and parameters from response, call the function with parameters. Append the result to messages. <br>    
**Step 4**: Invoke the chat completions API with the message list to get the response.


key: _node_content
string_value:
{"id_": "58379fcd-b29d-465f-aaad-0299dcdfa07f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c295668d-af35-4b34-badc-23f8e2e5af33", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd0712c8da5d6b6f219f0c56c660eb55b7fb9a327e93f2f6093e7e01181ecff9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ccedcea-5c16-4a52-ac0b-f39355a38224", "node_type": "1", "metadata": {}, "hash": "07084e56be7dfbd48cf8bbd2718c15bf668cf6b93146d59675d4b4f241455355", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14831, "end_char_idx": 17906, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
tools = [
    {
        "type": "function",
        "function": {
            "name": "ask_database",
            "description": "Use this function to answer user questions about music. Input should be a fully formed SQL query.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": f"""
                                SQL query extracting info to answer the user's question.
                                SQL should be written using this database schema:
                                {database_schema_string}
                                The query should be returned in plain text, not in JSON.
                                """,
                    }
                },
                "required": ["query"],
            },
        }
    }
]
```

### Executing SQL queries

Now let's implement the function that will actually excute queries against the database.


```python
def ask_database(conn, query):
    """Function to query SQLite database with a provided SQL query."""
    try:
        results = str(conn.execute(query).fetchall())
    except Exception as e:
        results = f"query failed with error: {e}"
    return results
```

##### Steps to invoke a function call using Chat Completions API: 

**Step 1**: Prompt the model with content that may result in model selecting a tool to use. The description of the tools such as a function names and signature is defined in the 'Tools' list and passed to the model in API call. If selected, the function name and parameters are included in the response.<br>
  
**Step 2**: Check programmatically if model wanted to call a function. If true, proceed to step 3. <br>  
**Step 3**: Extract the function name and parameters from response, call the function with parameters. Append the result to messages. <br>    
**Step 4**: Invoke the chat completions API with the message list to get the response. 


```python
# Step #1: Prompt with content that may result in function call. In this case the model can identify the information requested by the user is potentially available in the database schema passed to the model in Tools description. 
messages = [{
    "role":"user", 
    "content": "What is the name of the album with the most tracks?"
}]

response = client.chat.completions.create(
    model='gpt-4o', 
    messages=messages, 
    tools= tools, 
    tool_choice="auto"
)

# Append the message to messages list
response_message = response.choices[0].message 
messages.append(response_message)

print(response_message)
```

    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_wDN8uLjq2ofuU6rVx1k8Gw0e', function=Function(arguments='{"query":"SELECT Album.Title, COUNT(Track.TrackId) AS TrackCount FROM Album INNER JOIN Track ON Album.AlbumId = Track.AlbumId GROUP BY Album.Title ORDER BY TrackCount DESC LIMIT 1;"}', name='ask_database'), type='function')])


key: _node_content
string_value:
{"id_": "9ccedcea-5c16-4a52-ac0b-f39355a38224", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3ec27c92-4f2e-4676-9611-ace050959b1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "465755c3f6b9fdf61cef5225b387867709ba20e7037a6e567db3b085ed7f5cd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58379fcd-b29d-465f-aaad-0299dcdfa07f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb", "title": "How to call functions with chat models", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5b1448a1e489b6eeac6b744eb560804d301a4b87b8fb287e9b19690b293775c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16866, "end_char_idx": 19767, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Step #1: Prompt with content that may result in function call. In this case the model can identify the information requested by the user is potentially available in the database schema passed to the model in Tools description. 
messages = [{
    "role":"user", 
    "content": "What is the name of the album with the most tracks?"
}]

response = client.chat.completions.create(
    model='gpt-4o', 
    messages=messages, 
    tools= tools, 
    tool_choice="auto"
)

# Append the message to messages list
response_message = response.choices[0].message 
messages.append(response_message)

print(response_message)
```

    ChatCompletionMessage(content=None, role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_wDN8uLjq2ofuU6rVx1k8Gw0e', function=Function(arguments='{"query":"SELECT Album.Title, COUNT(Track.TrackId) AS TrackCount FROM Album INNER JOIN Track ON Album.AlbumId = Track.AlbumId GROUP BY Album.Title ORDER BY TrackCount DESC LIMIT 1;"}', name='ask_database'), type='function')])



```python
# Step 2: determine if the response from the model includes a tool call.   
tool_calls = response_message.tool_calls
if tool_calls:
    # If true the model will return the name of the tool / function to call and the argument(s)  
    tool_call_id = tool_calls[0].id
    tool_function_name = tool_calls[0].function.name
    tool_query_string = json.loads(tool_calls[0].function.arguments)['query']

    # Step 3: Call the function and retrieve results. Append the results to the messages list.      
    if tool_function_name == 'ask_database':
        results = ask_database(conn, tool_query_string)
        
        messages.append({
            "role":"tool", 
            "tool_call_id":tool_call_id, 
            "name": tool_function_name, 
            "content":results
        })
        
        # Step 4: Invoke the chat completions API with the function response appended to the messages list
        # Note that messages with role 'tool' must be a response to a preceding message with 'tool_calls'
        model_response_with_function_call = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
        )  # get a new response from the model where it can see the function response
        print(model_response_with_function_call.choices[0].message.content)
    else: 
        print(f"Error: function {tool_function_name} does not exist")
else: 
    # Model did not identify a function to call, result can be returned to the user 
    print(response_message.content) 
```

    The album with the most tracks is titled "Greatest Hits," which contains 57 tracks.


## Next Steps

See our other [notebook](How_to_call_functions_for_knowledge_retrieval.ipynb) that demonstrates how to use the Chat Completions API and functions for knowledge retrieval to interact conversationally with a knowledge base.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb


key: _node_content
string_value:
{"id_": "bc91f25b-7d85-4415-8824-fcab754b72ef", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6abaf9d0-7d46-47c4-82c4-5ef7003a83ac", "node_type": "1", "metadata": {}, "hash": "ca1abd482282c8464f2b2e19ea362028f7b844b8c6a27ddeabe6f63a2ddaa246", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3146, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to build an agent with the Node.js SDK

OpenAI functions enable your app to take action based on user inputs. This means that it can, e.g., search the web, send emails, or book tickets on behalf of your users, making it more powerful than a regular chatbot.

In this tutorial, you will build an app that uses OpenAI functions along with the latest version of the Node.js SDK. The app runs in the browser, so you only need a code editor and, e.g., VS Code Live Server to follow along locally. Alternatively, write your code directly in the browser via [this code playground at Scrimba.](https://scrimba.com/scrim/c6r3LkU9)

## What you will build

Our app is a simple agent that helps you find activities in your area.
It has access to two functions, `getLocation()` and `getCurrentWeather()`,
which means it can figure out where you’re located and what the weather
is at the moment.

At this point, it's important to understand that
OpenAI doesn't execute any code for you. It just tells your app which
functions it should use in a given scenario, and then leaves it up to
your app to invoke them.

Once our agent knows your location and the weather, it'll use GPT’s
internal knowledge to suggest suitable local activities for you.

## Importing the SDK and authenticating with OpenAI

We start by importing the OpenAI SDK at the top of our JavaScript file and authenticate with our API key, which we have stored as an environment variable.

```js
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  dangerouslyAllowBrowser: true,
});
```

Since we're running our code in a browser environment at Scrimba, we also need to set `dangerouslyAllowBrowser: true` to confirm we understand the risks involved with client-side API requests. Please note that you should move these requests over to a Node server in a production app.

## Creating our two functions

Next, we'll create the two functions. The first one - `getLocation` -
uses the [IP API](https://ipapi.co/) to get the location of the
user.

```js
async function getLocation() {
  const response = await fetch("https://ipapi.co/json/");
  const locationData = await response.json();
  return locationData;
}
```

The IP API returns a bunch of data about your location, including your
latitude and longitude, which we’ll use as arguments in the second
function `getCurrentWeather`. It uses the [Open Meteo
API](https://open-meteo.com/) to get the current weather data, like
this:

```js
async function getCurrentWeather(latitude, longitude) {
  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
  const response = await fetch(url);
  const weatherData = await response.json();
  return weatherData;
}
```

## Describing our functions for OpenAI

For OpenAI to understand the purpose of these functions, we need to
describe them using a specific schema. We'll create an array called
`tools` that contains one object per function. Each object
will have two keys: `type`, `function`, and the `function` key has 
three subkeys: `name`, `description`, and `parameters`.


key: _node_content
string_value:
{"id_": "6abaf9d0-7d46-47c4-82c4-5ef7003a83ac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc91f25b-7d85-4415-8824-fcab754b72ef", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "29c9e1c00d9639c324b06f2a936f7bc6e61d84dcb17986a83f1bee4a2bf5160f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c8e895d-2d12-40ed-af5e-4ebbad73e7ae", "node_type": "1", "metadata": {}, "hash": "ca2e5d26e7b9ee64c4d5aafca21d6281e27eabbde2460e570eb871d0b9c40752", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1785, "end_char_idx": 4631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Please note that you should move these requests over to a Node server in a production app.

## Creating our two functions

Next, we'll create the two functions. The first one - `getLocation` -
uses the [IP API](https://ipapi.co/) to get the location of the
user.

```js
async function getLocation() {
  const response = await fetch("https://ipapi.co/json/");
  const locationData = await response.json();
  return locationData;
}
```

The IP API returns a bunch of data about your location, including your
latitude and longitude, which we’ll use as arguments in the second
function `getCurrentWeather`. It uses the [Open Meteo
API](https://open-meteo.com/) to get the current weather data, like
this:

```js
async function getCurrentWeather(latitude, longitude) {
  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
  const response = await fetch(url);
  const weatherData = await response.json();
  return weatherData;
}
```

## Describing our functions for OpenAI

For OpenAI to understand the purpose of these functions, we need to
describe them using a specific schema. We'll create an array called
`tools` that contains one object per function. Each object
will have two keys: `type`, `function`, and the `function` key has 
three subkeys: `name`, `description`, and `parameters`.

```js
const tools = [
  {
    type: "function",
    function: {
      name: "getCurrentWeather",
      description: "Get the current weather in a given location",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
          },
          longitude: {
            type: "string",
          },
        },
        required: ["longitude", "latitude"],
      },
    }
  },
  {
    type: "function",
    function: {
      name: "getLocation",
      description: "Get the user's location based on their IP address",
      parameters: {
        type: "object",
        properties: {},
      },
    }
  },
];
```

## Setting up the messages array

We also need to define a `messages` array. This will keep track of all of the messages back and forth between our app and OpenAI.

The first object in the array should always have the `role` property set to `"system"`, which tells OpenAI that this is how we want it to behave.

```js
const messages = [
  {
    role: "system",
    content:
      "You are a helpful assistant. Only use the functions you have been provided with.",
  },
];
```

## Creating the agent function

We are now ready to build the logic of our app, which lives in the
`agent` function. It is asynchronous and takes one argument: the
`userInput`.

We start by pushing the `userInput` to the messages array. This time, we set the `role` to `"user"`, so that OpenAI knows that this is the input from the user.


key: _node_content
string_value:
{"id_": "3c8e895d-2d12-40ed-af5e-4ebbad73e7ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6abaf9d0-7d46-47c4-82c4-5ef7003a83ac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ee2d3932082a044e89551236f11a6257f9839d1897bb4991db947f7d60084290", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbd8d90c-0154-4c1b-b267-32813364213a", "node_type": "1", "metadata": {}, "hash": "a5196a1bcd36d11233917bb7cf99623661cf52416c626958c07e77776b086e64", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3011, "end_char_idx": 5625, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Each object
will have two keys: `type`, `function`, and the `function` key has 
three subkeys: `name`, `description`, and `parameters`.

```js
const tools = [
  {
    type: "function",
    function: {
      name: "getCurrentWeather",
      description: "Get the current weather in a given location",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
          },
          longitude: {
            type: "string",
          },
        },
        required: ["longitude", "latitude"],
      },
    }
  },
  {
    type: "function",
    function: {
      name: "getLocation",
      description: "Get the user's location based on their IP address",
      parameters: {
        type: "object",
        properties: {},
      },
    }
  },
];
```

## Setting up the messages array

We also need to define a `messages` array. This will keep track of all of the messages back and forth between our app and OpenAI.

The first object in the array should always have the `role` property set to `"system"`, which tells OpenAI that this is how we want it to behave.

```js
const messages = [
  {
    role: "system",
    content:
      "You are a helpful assistant. Only use the functions you have been provided with.",
  },
];
```

## Creating the agent function

We are now ready to build the logic of our app, which lives in the
`agent` function. It is asynchronous and takes one argument: the
`userInput`.

We start by pushing the `userInput` to the messages array. This time, we set the `role` to `"user"`, so that OpenAI knows that this is the input from the user.

```js
async function agent(userInput) {
  messages.push({
    role: "user",
    content: userInput,
  });
  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: messages,
    tools: tools,
  });
  console.log(response);
}
```

Next, we'll send a request to the Chat completions endpoint via the
`chat.completions.create()` method in the Node SDK. This method takes a
configuration object as an argument. In it, we'll specify three
properties:

- `model` - Decides which AI model we want to use (in our case,
  GPT-4).
- `messages` - The entire history of messages between the user and the
  AI up until this point.
- `tools` - A list of tools the model may call. Currently, only 
  functions are supported as a tool., we'll we use the `tools` array we 
  created earlier.

## Running our app with a simple input

Let's try to run the `agent` with an input that requires a function call to give a suitable reply.

```js
agent("Where am I located right now?


key: _node_content
string_value:
{"id_": "fbd8d90c-0154-4c1b-b267-32813364213a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c8e895d-2d12-40ed-af5e-4ebbad73e7ae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f16ff4f48aa6f9b0d612b32ad40e6e1447fef323769b7d64f19e084c9b7dc447", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03a5d7f8-5dd6-459d-b224-5107f87e2550", "node_type": "1", "metadata": {}, "hash": "156ab0952c55e053cf558d83723ada32eaeede13fd629274da1d9f9cb20b9b8a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4129, "end_char_idx": 6904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```js
const messages = [
  {
    role: "system",
    content:
      "You are a helpful assistant. Only use the functions you have been provided with.",
  },
];
```

## Creating the agent function

We are now ready to build the logic of our app, which lives in the
`agent` function. It is asynchronous and takes one argument: the
`userInput`.

We start by pushing the `userInput` to the messages array. This time, we set the `role` to `"user"`, so that OpenAI knows that this is the input from the user.

```js
async function agent(userInput) {
  messages.push({
    role: "user",
    content: userInput,
  });
  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: messages,
    tools: tools,
  });
  console.log(response);
}
```

Next, we'll send a request to the Chat completions endpoint via the
`chat.completions.create()` method in the Node SDK. This method takes a
configuration object as an argument. In it, we'll specify three
properties:

- `model` - Decides which AI model we want to use (in our case,
  GPT-4).
- `messages` - The entire history of messages between the user and the
  AI up until this point.
- `tools` - A list of tools the model may call. Currently, only 
  functions are supported as a tool., we'll we use the `tools` array we 
  created earlier.

## Running our app with a simple input

Let's try to run the `agent` with an input that requires a function call to give a suitable reply.

```js
agent("Where am I located right now?");
```

When we run the code above, we see the response from OpenAI logged out
to the console like this:

```js
{
    id: "chatcmpl-84ojoEJtyGnR6jRHK2Dl4zTtwsa7O",
    object: "chat.completion",
    created: 1696159040,
    model: "gpt-4-0613",
    choices: [{
        index: 0,
        message: {
            role: "assistant",
            content: null,
            tool_calls: [
              id: "call_CBwbo9qoXUn1kTR5pPuv6vR1",
              type: "function",
              function: {
                name: "getLocation",
                arguments: "{}"
              }
            ]
        },
        logprobs: null,
        finish_reason: "tool_calls" // OpenAI wants us to call a function
    }],
    usage: {
        prompt_tokens: 134,
        completion_tokens: 6,
        total_tokens: 140
    }
     system_fingerprint: null
}
```

This response tells us that we should call one of our functions, as it contains the following key: `finish_reason: "tool_calls"`.

The name of the function can be found in the
`response.choices[0].message.tool_calls[0].function.name` key, which is set to
`"getLocation"`.

## Turning the OpenAI response into a function call

Now that we have the name of the function as a string, we'll need to
translate that into a function call.


key: _node_content
string_value:
{"id_": "03a5d7f8-5dd6-459d-b224-5107f87e2550", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbd8d90c-0154-4c1b-b267-32813364213a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8b8ec9f28d3b71f06f5efe7ae132799581ae3afd5c24775c9d3d4b16a15fb123", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e0f32e0-4599-4ca2-9523-c4c8a9033293", "node_type": "1", "metadata": {}, "hash": "5bac127fd4eb5b14b3f065fd46d55d20acac5b808a38b458e7e9f841674a8fbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5443, "end_char_idx": 8167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Running our app with a simple input

Let's try to run the `agent` with an input that requires a function call to give a suitable reply.

```js
agent("Where am I located right now?");
```

When we run the code above, we see the response from OpenAI logged out
to the console like this:

```js
{
    id: "chatcmpl-84ojoEJtyGnR6jRHK2Dl4zTtwsa7O",
    object: "chat.completion",
    created: 1696159040,
    model: "gpt-4-0613",
    choices: [{
        index: 0,
        message: {
            role: "assistant",
            content: null,
            tool_calls: [
              id: "call_CBwbo9qoXUn1kTR5pPuv6vR1",
              type: "function",
              function: {
                name: "getLocation",
                arguments: "{}"
              }
            ]
        },
        logprobs: null,
        finish_reason: "tool_calls" // OpenAI wants us to call a function
    }],
    usage: {
        prompt_tokens: 134,
        completion_tokens: 6,
        total_tokens: 140
    }
     system_fingerprint: null
}
```

This response tells us that we should call one of our functions, as it contains the following key: `finish_reason: "tool_calls"`.

The name of the function can be found in the
`response.choices[0].message.tool_calls[0].function.name` key, which is set to
`"getLocation"`.

## Turning the OpenAI response into a function call

Now that we have the name of the function as a string, we'll need to
translate that into a function call. To help us with that, we'll gather
both of our functions in an object called `availableTools`:

```js
const availableTools = {
  getCurrentWeather,
  getLocation,
};
```

This is handy because we'll be able to access the `getLocation` function
via bracket notation and the string we got back from OpenAI, like this:
`availableTools["getLocation"]`.

```js
const { finish_reason, message } = response.choices[0];

if (finish_reason === "tool_calls" && message.tool_calls) {
  const functionName = message.tool_calls[0].function.name;
  const functionToCall = availableTools[functionName];
  const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
  const functionArgsArr = Object.values(functionArgs);
  const functionResponse = await functionToCall.apply(null, functionArgsArr);
  console.log(functionResponse);
}
```

We're also grabbing ahold of any arguments OpenAI wants us to pass into
the function: `message.tool_calls[0].function.arguments`.
However, we won't need any arguments for this first function call.

If we run the code again with the same input
(`"Where am I located right now?"`), we'll see that `functionResponse`
is an object filled with location about where the user is located right
now. In my case, that is Oslo, Norway.


key: _node_content
string_value:
{"id_": "3e0f32e0-4599-4ca2-9523-c4c8a9033293", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03a5d7f8-5dd6-459d-b224-5107f87e2550", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a50d3d6443358666831d3311be386983b2f71fa1c6a791e7ee175d1b597bb27", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cba7a6ad-bac0-45ce-a56d-7faaa70c2084", "node_type": "1", "metadata": {}, "hash": "4acfcc5b2f43f077fca35e309599e288082196449b9519ce1e67e1c153393778", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6604, "end_char_idx": 9553, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The name of the function can be found in the
`response.choices[0].message.tool_calls[0].function.name` key, which is set to
`"getLocation"`.

## Turning the OpenAI response into a function call

Now that we have the name of the function as a string, we'll need to
translate that into a function call. To help us with that, we'll gather
both of our functions in an object called `availableTools`:

```js
const availableTools = {
  getCurrentWeather,
  getLocation,
};
```

This is handy because we'll be able to access the `getLocation` function
via bracket notation and the string we got back from OpenAI, like this:
`availableTools["getLocation"]`.

```js
const { finish_reason, message } = response.choices[0];

if (finish_reason === "tool_calls" && message.tool_calls) {
  const functionName = message.tool_calls[0].function.name;
  const functionToCall = availableTools[functionName];
  const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
  const functionArgsArr = Object.values(functionArgs);
  const functionResponse = await functionToCall.apply(null, functionArgsArr);
  console.log(functionResponse);
}
```

We're also grabbing ahold of any arguments OpenAI wants us to pass into
the function: `message.tool_calls[0].function.arguments`.
However, we won't need any arguments for this first function call.

If we run the code again with the same input
(`"Where am I located right now?"`), we'll see that `functionResponse`
is an object filled with location about where the user is located right
now. In my case, that is Oslo, Norway.

```js
{ip: "193.212.60.170", network: "193.212.60.0/23", version: "IPv4", city: "Oslo", region: "Oslo County", region_code: "03", country: "NO", country_name: "Norway", country_code: "NO", country_code_iso3: "NOR", country_capital: "Oslo", country_tld: ".no", continent_code: "EU", in_eu: false, postal: "0026", latitude: 59.955, longitude: 10.859, timezone: "Europe/Oslo", utc_offset: "+0200", country_calling_code: "+47", currency: "NOK", currency_name: "Krone", languages: "no,nb,nn,se,fi", country_area: 324220, country_population: 5314336, asn: "AS2119", org: "Telenor Norge AS"}
```

We'll add this data to a new item in the `messages` array, where we also
specify the name of the function we called.

```js
messages.push({
  role: "function",
  name: functionName,
  content: `The result of the last function was this: ${JSON.stringify(
    functionResponse
  )}
  `,
});
```

Notice that the `role` is set to `"function"`. This tells OpenAI
that the `content` parameter contains the result of the function call
and not the input from the user.

At this point, we need to send a new request to OpenAI with this updated
`messages` array. However, we don’t want to hard code a new function
call, as our agent might need to go back and forth between itself and
GPT several times until it has found the final answer for the user.

This can be solved in several different ways, e.g.


key: _node_content
string_value:
{"id_": "cba7a6ad-bac0-45ce-a56d-7faaa70c2084", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e0f32e0-4599-4ca2-9523-c4c8a9033293", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9fd205e51fdc3727ab7f5172b80bf09525fc996eaf6ce84f95cdc7b2fe5b6861", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17aed5f6-f2fe-427b-8807-2a123724d149", "node_type": "1", "metadata": {}, "hash": "b1885671f8b1758a1018aaa2f5bff352cb9ead17165d574b27a5892209cfc6f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8877, "end_char_idx": 12087, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```js
messages.push({
  role: "function",
  name: functionName,
  content: `The result of the last function was this: ${JSON.stringify(
    functionResponse
  )}
  `,
});
```

Notice that the `role` is set to `"function"`. This tells OpenAI
that the `content` parameter contains the result of the function call
and not the input from the user.

At this point, we need to send a new request to OpenAI with this updated
`messages` array. However, we don’t want to hard code a new function
call, as our agent might need to go back and forth between itself and
GPT several times until it has found the final answer for the user.

This can be solved in several different ways, e.g. recursion, a
while-loop, or a for-loop. We'll use a good old for-loop for the sake of
simplicity.

## Creating the loop

At the top of the `agent` function, we'll create a loop that lets us run
the entire procedure up to five times.

If we get back `finish_reason: "tool_calls"` from GPT, we'll just
push the result of the function call to the `messages` array and jump to
the next iteration of the loop, triggering a new request.

If we get `finish_reason: "stop"` back, then GPT has found a suitable
answer, so we'll return the function and cancel the loop.

```js
for (let i = 0; i < 5; i++) {
  const response = await openai.chat.completions.create({
    model: "gpt-4",
    messages: messages,
    tools: tools,
  });
  const { finish_reason, message } = response.choices[0];

  if (finish_reason === "tool_calls" && message.tool_calls) {
    const functionName = message.tool_calls[0].function.name;
    const functionToCall = availableTools[functionName];
    const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
    const functionArgsArr = Object.values(functionArgs);
    const functionResponse = await functionToCall.apply(null, functionArgsArr);

    messages.push({
      role: "function",
      name: functionName,
      content: `
          The result of the last function was this: ${JSON.stringify(
            functionResponse
          )}
          `,
    });
  } else if (finish_reason === "stop") {
    messages.push(message);
    return message.content;
  }
}
return "The maximum number of iterations has been met without a suitable answer. Please try again with a more specific input.";
```

If we don't see a `finish_reason: "stop"` within our five iterations,
we'll return a message saying we couldn’t find a suitable answer.

## Running the final app

At this point, we are ready to try our app! I'll ask the agent to
suggest some activities based on my location and the current weather.

```js
const response = await agent(
  "Please suggest some activities based on my location and the current weather."
);
console.log(response);
```

Here's what we see in the console (formatted to make it easier to read):

```js
Based on your current location in Oslo, Norway and the weather (15°C and snowy),
here are some activity suggestions:

1. A visit to the Oslo Winter Park for skiing or snowboarding.
2. Enjoy a cosy day at a local café or restaurant.
3. Visit one of Oslo's many museums. The Fram Museum or Viking Ship Museum offer interesting insights into Norway’s seafaring history.
4.


key: _node_content
string_value:
{"id_": "17aed5f6-f2fe-427b-8807-2a123724d149", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cba7a6ad-bac0-45ce-a56d-7faaa70c2084", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cb06b31432e5f60fb9cdb1dd668602442b78a35ce9d02b0b67894faac8b2ce37", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88d651df-742c-489c-8821-8acd43e5f474", "node_type": "1", "metadata": {}, "hash": "878dc08538da8f65fe66c6a3dee4de13f8bb480fe45b4cf715ad1ce893704a08", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11138, "end_char_idx": 13479, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Please try again with a more specific input.";
```

If we don't see a `finish_reason: "stop"` within our five iterations,
we'll return a message saying we couldn’t find a suitable answer.

## Running the final app

At this point, we are ready to try our app! I'll ask the agent to
suggest some activities based on my location and the current weather.

```js
const response = await agent(
  "Please suggest some activities based on my location and the current weather."
);
console.log(response);
```

Here's what we see in the console (formatted to make it easier to read):

```js
Based on your current location in Oslo, Norway and the weather (15°C and snowy),
here are some activity suggestions:

1. A visit to the Oslo Winter Park for skiing or snowboarding.
2. Enjoy a cosy day at a local café or restaurant.
3. Visit one of Oslo's many museums. The Fram Museum or Viking Ship Museum offer interesting insights into Norway’s seafaring history.
4. Take a stroll in the snowy streets and enjoy the beautiful winter landscape.
5. Enjoy a nice book by the fireplace in a local library.
6. Take a fjord sightseeing cruise to enjoy the snowy landscapes.

Always remember to bundle up and stay warm. Enjoy your day!
```

If we peak under the hood, and log out `response.choices[0].message` in
each iteration of the loop, we'll see that GPT has instructed us to use
both our functions before coming up with an answer.

First, it tells us to call the `getLocation` function. Then it tells us
to call the `getCurrentWeather` function with
`"longitude": "10.859", "latitude": "59.955"` passed in as the
arguments. This is data it got back from the first function call we did.

```js
{"role":"assistant","content":null,"tool_calls":[{"id":"call_Cn1KH8mtHQ2AMbyNwNJTweEP","type":"function","function":{"name":"getLocation","arguments":"{}"}}]}
{"role":"assistant","content":null,"tool_calls":[{"id":"call_uc1oozJfGTvYEfIzzcsfXfOl","type":"function","function":{"name":"getCurrentWeather","arguments":"{\n\"latitude\": \"10.859\",\n\"longitude\": \"59.955\"\n}"}}]}
```

You've now built an AI agent using OpenAI functions and the Node.js SDK! If you're looking for an extra challenge, consider enhancing this app. For example, you could add a function that fetches up-to-date information on events and activities in the user's location.

Happy coding!


key: _node_content
string_value:
{"id_": "88d651df-742c-489c-8821-8acd43e5f474", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17aed5f6-f2fe-427b-8807-2a123724d149", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8d81de62fea77afac1f7b7f1848622bba8971a6f09d4266096225a8ea4606619", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb6da9e7-671b-44c9-82e7-96bc96e386b0", "node_type": "1", "metadata": {}, "hash": "f7865ffd704d0cba63385c58a1801b60b329c1eb19d631c990a51a26b8af936f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11950, "end_char_idx": 14926, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Visit one of Oslo's many museums. The Fram Museum or Viking Ship Museum offer interesting insights into Norway’s seafaring history.
4. Take a stroll in the snowy streets and enjoy the beautiful winter landscape.
5. Enjoy a nice book by the fireplace in a local library.
6. Take a fjord sightseeing cruise to enjoy the snowy landscapes.

Always remember to bundle up and stay warm. Enjoy your day!
```

If we peak under the hood, and log out `response.choices[0].message` in
each iteration of the loop, we'll see that GPT has instructed us to use
both our functions before coming up with an answer.

First, it tells us to call the `getLocation` function. Then it tells us
to call the `getCurrentWeather` function with
`"longitude": "10.859", "latitude": "59.955"` passed in as the
arguments. This is data it got back from the first function call we did.

```js
{"role":"assistant","content":null,"tool_calls":[{"id":"call_Cn1KH8mtHQ2AMbyNwNJTweEP","type":"function","function":{"name":"getLocation","arguments":"{}"}}]}
{"role":"assistant","content":null,"tool_calls":[{"id":"call_uc1oozJfGTvYEfIzzcsfXfOl","type":"function","function":{"name":"getCurrentWeather","arguments":"{\n\"latitude\": \"10.859\",\n\"longitude\": \"59.955\"\n}"}}]}
```

You've now built an AI agent using OpenAI functions and the Node.js SDK! If you're looking for an extra challenge, consider enhancing this app. For example, you could add a function that fetches up-to-date information on events and activities in the user's location.

Happy coding!

<details>
<summary>Complete code</summary>

```js
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  dangerouslyAllowBrowser: true,
});

async function getLocation() {
  const response = await fetch("https://ipapi.co/json/");
  const locationData = await response.json();
  return locationData;
}

async function getCurrentWeather(latitude, longitude) {
  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
  const response = await fetch(url);
  const weatherData = await response.json();
  return weatherData;
}

const tools = [
  {
    type: "function",
    function: {
      name: "getCurrentWeather",
      description: "Get the current weather in a given location",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
          },
          longitude: {
            type: "string",
          },
        },
        required: ["longitude", "latitude"],
      },
    }
  },
  {
    type: "function",
    function: {
      name: "getLocation",
      description: "Get the user's location based on their IP address",
      parameters: {
        type: "object",
        properties: {},
      },
    }
  },
];

const availableTools = {
  getCurrentWeather,
  getLocation,
};

const messages = [
  {
    role: "system",
    content: `You are a helpful assistant.


key: _node_content
string_value:
{"id_": "bb6da9e7-671b-44c9-82e7-96bc96e386b0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "500b70a0-f269-4bd3-b262-69087a3cba74", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9e4129f15ce147be64461e7b3f4cece759d4a6614b81e7d5afbe434e9b452d3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88d651df-742c-489c-8821-8acd43e5f474", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_an_agent_with_the_node_sdk.ipynb", "title": "How to build an agent with the Node.js SDK", "tokens": 3947, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "731fde036ed5f91290f363b28a9caa3c916b8177d1a60ba782830ec793a955e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13271, "end_char_idx": 16413, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you're looking for an extra challenge, consider enhancing this app. For example, you could add a function that fetches up-to-date information on events and activities in the user's location.

Happy coding!

<details>
<summary>Complete code</summary>

```js
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  dangerouslyAllowBrowser: true,
});

async function getLocation() {
  const response = await fetch("https://ipapi.co/json/");
  const locationData = await response.json();
  return locationData;
}

async function getCurrentWeather(latitude, longitude) {
  const url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=apparent_temperature`;
  const response = await fetch(url);
  const weatherData = await response.json();
  return weatherData;
}

const tools = [
  {
    type: "function",
    function: {
      name: "getCurrentWeather",
      description: "Get the current weather in a given location",
      parameters: {
        type: "object",
        properties: {
          latitude: {
            type: "string",
          },
          longitude: {
            type: "string",
          },
        },
        required: ["longitude", "latitude"],
      },
    }
  },
  {
    type: "function",
    function: {
      name: "getLocation",
      description: "Get the user's location based on their IP address",
      parameters: {
        type: "object",
        properties: {},
      },
    }
  },
];

const availableTools = {
  getCurrentWeather,
  getLocation,
};

const messages = [
  {
    role: "system",
    content: `You are a helpful assistant. Only use the functions you have been provided with.`,
  },
];

async function agent(userInput) {
  messages.push({
    role: "user",
    content: userInput,
  });

  for (let i = 0; i < 5; i++) {
    const response = await openai.chat.completions.create({
      model: "gpt-4",
      messages: messages,
      tools: tools,
    });

    const { finish_reason, message } = response.choices[0];

    if (finish_reason === "tool_calls" && message.tool_calls) {
      const functionName = message.tool_calls[0].function.name;
      const functionToCall = availableTools[functionName];
      const functionArgs = JSON.parse(message.tool_calls[0].function.arguments);
      const functionArgsArr = Object.values(functionArgs);
      const functionResponse = await functionToCall.apply(
        null,
        functionArgsArr
      );

      messages.push({
        role: "function",
        name: functionName,
        content: `
                The result of the last function was this: ${JSON.stringify(
                  functionResponse
                )}
                `,
      });
    } else if (finish_reason === "stop") {
      messages.push(message);
      return message.content;
    }
  }
  return "The maximum number of iterations has been met without a suitable answer. Please try again with a more specific input.";
}

const response = await agent(
  "Please suggest some activities based on my location and the weather."
);

console.log("response:", response);
```

</details>


key: _node_content
string_value:
{"id_": "7aa9d898-dc38-4ba9-a1cc-06a77fd7077d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4976cd51-d59f-4b98-a5c9-ac2d72a5f52f", "node_type": "1", "metadata": {}, "hash": "ce2a580a5afd89fd71b66665fe1b4b24f3e9d9034b478faeaf45c0e25150c635", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3155, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Enhancing Whisper transcriptions: pre- & post-processing techniques

This notebook offers a guide to improve the Whisper's transcriptions. We'll streamline your audio data via trimming and segmentation, enhancing Whisper's transcription quality. After transcriptions, we'll refine the output by adding punctuation, adjusting product terminology (e.g., 'five two nine' to '529'), and mitigating Unicode issues. These strategies will help improve the clarity of your transcriptions, but remember, customization based on your unique use-case may be beneficial.



## Setup

To get started let's import a few different libraries:

- [PyDub](http://pydub.com/) is a simple and easy-to-use Python library for audio processing tasks such as slicing, concatenating, and exporting audio files.

- The `Audio` class from the `IPython.display` module allows you to create an audio control that can play sound in Jupyter notebooks, providing a straightforward way to play audio data directly in your notebook.

- For our audio file, we'll use a fictional earnings call written by ChatGPT and read aloud by the author.This audio file is relatively short, but hopefully provides you with an illustrative idea of how these pre and post processing steps can be applied to any audio file. 


```python
from openai import OpenAI
import os
import urllib
from IPython.display import Audio
from pathlib import Path
from pydub import AudioSegment
import ssl
```


```python
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```


```python
# set download paths
earnings_call_remote_filepath = "https://cdn.openai.com/API/examples/data/EarningsCall.wav"

# set local save locations
earnings_call_filepath = "data/EarningsCall.wav"

# download example audio files and save locally
ssl._create_default_https_context = ssl._create_unverified_context
urllib.request.urlretrieve(earnings_call_remote_filepath, earnings_call_filepath)
```




    ('data/EarningsCall.wav', <http.client.HTTPMessage at 0x11be41f50>)



At times, files with long silences at the beginning can cause Whisper to transcribe the audio incorrectly. We'll use Pydub to detect and trim the silence. 

Here, we've set the decibel threshold of 20. You can change this if you would like.


```python
# Function to detect leading silence
# Returns the number of milliseconds until the first sound (chunk averaging more than X decibels)
def milliseconds_until_sound(sound, silence_threshold_in_decibels=-20.0, chunk_size=10):
    trim_ms = 0  # ms

    assert chunk_size > 0  # to avoid infinite loop
    while sound[trim_ms:trim_ms+chunk_size].dBFS < silence_threshold_in_decibels and trim_ms < len(sound):
        trim_ms += chunk_size

    return trim_ms

```


```python
def trim_start(filepath):
    path = Path(filepath)
    directory = path.parent
    filename = path.name
    audio = AudioSegment.from_file(filepath, format="wav")
    start_trim = milliseconds_until_sound(audio)
    trimmed = audio[start_trim:]
    new_filename = directory / f"trimmed_{filename}"
    trimmed.export(new_filename, format="wav")
    return trimmed, new_filename

```


key: _node_content
string_value:
{"id_": "4976cd51-d59f-4b98-a5c9-ac2d72a5f52f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7aa9d898-dc38-4ba9-a1cc-06a77fd7077d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a936cbdd5cf261d67d40e4b14c80564c3a0b7c8bc3c411b33d65442568f42125", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e40a706c-9192-4dc6-8334-5e66f345e9c4", "node_type": "1", "metadata": {}, "hash": "250343fa2b7446d5d05e80aa64a3da5a50a7ea2b7842df0bcafe033caa211721", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1565, "end_char_idx": 5033, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# set download paths
earnings_call_remote_filepath = "https://cdn.openai.com/API/examples/data/EarningsCall.wav"

# set local save locations
earnings_call_filepath = "data/EarningsCall.wav"

# download example audio files and save locally
ssl._create_default_https_context = ssl._create_unverified_context
urllib.request.urlretrieve(earnings_call_remote_filepath, earnings_call_filepath)
```




    ('data/EarningsCall.wav', <http.client.HTTPMessage at 0x11be41f50>)



At times, files with long silences at the beginning can cause Whisper to transcribe the audio incorrectly. We'll use Pydub to detect and trim the silence. 

Here, we've set the decibel threshold of 20. You can change this if you would like.


```python
# Function to detect leading silence
# Returns the number of milliseconds until the first sound (chunk averaging more than X decibels)
def milliseconds_until_sound(sound, silence_threshold_in_decibels=-20.0, chunk_size=10):
    trim_ms = 0  # ms

    assert chunk_size > 0  # to avoid infinite loop
    while sound[trim_ms:trim_ms+chunk_size].dBFS < silence_threshold_in_decibels and trim_ms < len(sound):
        trim_ms += chunk_size

    return trim_ms

```


```python
def trim_start(filepath):
    path = Path(filepath)
    directory = path.parent
    filename = path.name
    audio = AudioSegment.from_file(filepath, format="wav")
    start_trim = milliseconds_until_sound(audio)
    trimmed = audio[start_trim:]
    new_filename = directory / f"trimmed_{filename}"
    trimmed.export(new_filename, format="wav")
    return trimmed, new_filename

```


```python
def transcribe_audio(file,output_dir):
    audio_path = os.path.join(output_dir, file)
    with open(audio_path, 'rb') as audio_data:
        transcription = client.audio.transcriptions.create(
            model="whisper-1", file=audio_data)
        return transcription.text
```

At times, we've seen unicode character injection in transcripts, removing any non-ASCII characters should help mitigate this issue.

Keep in mind you should not use this function if you are transcribing in Greek, Cyrillic, Arabic, Chinese, etc


```python
# Define function to remove non-ascii characters
def remove_non_ascii(text):
    return ''.join(i for i in text if ord(i)<128)

```

This function will add formatting and punctuation to our transcript. Whisper generates a transcript with punctuation but without formatting.


```python
# Define function to add punctuation
def punctuation_assistant(ascii_transcript):

    system_prompt = """You are a helpful assistant that adds punctuation to text.
      Preserve the original words and only insert necessary punctuation such as periods,
     commas, capialization, symbols like dollar sings or percentage signs, and formatting.
     Use only the context provided. If there is no context provided say, 'No context provided'\n"""
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": ascii_transcript
            }
        ]
    )
    return response

```

Our audio file is a recording from a fake earnings call that includes a lot of financial products. This function can help ensure that if Whisper transcribes these financial product names incorrectly, that they can be corrected.


key: _node_content
string_value:
{"id_": "e40a706c-9192-4dc6-8334-5e66f345e9c4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4976cd51-d59f-4b98-a5c9-ac2d72a5f52f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "15ad041000a42a0f6d992a064ce0eca8be8b9fe7998ee0de8c45361226fba0a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b906d5a1-3a54-4b25-b149-e819d43a0a1f", "node_type": "1", "metadata": {}, "hash": "4e9490c34d3fe134717faef8ddbacf62f2eefefbeea076dcecf05bc590620988", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3158, "end_char_idx": 7246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def transcribe_audio(file,output_dir):
    audio_path = os.path.join(output_dir, file)
    with open(audio_path, 'rb') as audio_data:
        transcription = client.audio.transcriptions.create(
            model="whisper-1", file=audio_data)
        return transcription.text
```

At times, we've seen unicode character injection in transcripts, removing any non-ASCII characters should help mitigate this issue.

Keep in mind you should not use this function if you are transcribing in Greek, Cyrillic, Arabic, Chinese, etc


```python
# Define function to remove non-ascii characters
def remove_non_ascii(text):
    return ''.join(i for i in text if ord(i)<128)

```

This function will add formatting and punctuation to our transcript. Whisper generates a transcript with punctuation but without formatting.


```python
# Define function to add punctuation
def punctuation_assistant(ascii_transcript):

    system_prompt = """You are a helpful assistant that adds punctuation to text.
      Preserve the original words and only insert necessary punctuation such as periods,
     commas, capialization, symbols like dollar sings or percentage signs, and formatting.
     Use only the context provided. If there is no context provided say, 'No context provided'\n"""
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": ascii_transcript
            }
        ]
    )
    return response

```

Our audio file is a recording from a fake earnings call that includes a lot of financial products. This function can help ensure that if Whisper transcribes these financial product names incorrectly, that they can be corrected. 


```python
# Define function to fix product mispellings
def product_assistant(ascii_transcript):
    system_prompt = """You are an intelligent assistant specializing in financial products;
    your task is to process transcripts of earnings calls, ensuring that all references to
     financial products and common financial terms are in the correct format. For each
     financial product or common term that is typically abbreviated as an acronym, the full term 
    should be spelled out followed by the acronym in parentheses. For example, '401k' should be
     transformed to '401(k) retirement savings plan', 'HSA' should be transformed to 'Health Savings Account (HSA)'
    , 'ROA' should be transformed to 'Return on Assets (ROA)', 'VaR' should be transformed to 'Value at Risk (VaR)'
, and 'PB' should be transformed to 'Price to Book (PB) ratio'. Similarly, transform spoken numbers representing 
financial products into their numeric representations, followed by the full name of the product in parentheses. 
For instance, 'five two nine' to '529 (Education Savings Plan)' and 'four zero one k' to '401(k) (Retirement Savings Plan)'.
 However, be aware that some acronyms can have different meanings based on the context (e.g., 'LTV' can stand for 
'Loan to Value' or 'Lifetime Value'). You will need to discern from the context which term is being referred to 
and apply the appropriate transformation. In cases where numerical figures or metrics are spelled out but do not 
represent specific financial products (like 'twenty three percent'), these should be left as is. Your role is to
 analyze and adjust financial product terminology in the text. Once you've done that, produce the adjusted 
 transcript and a list of the words you've changed"""
    response = client.chat.completions.create(
        model="gpt-4",
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": ascii_transcript
            }
        ]
    )
    return response

```

This function will create a new file with 'trimmed' appended to the original file name


key: _node_content
string_value:
{"id_": "b906d5a1-3a54-4b25-b149-e819d43a0a1f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e40a706c-9192-4dc6-8334-5e66f345e9c4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "629dc50165c993ec9ea5abee10e2d485c52531aa76e361f0cf77f076506045ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c707fd65-16fa-4add-89d7-84da12469f5e", "node_type": "1", "metadata": {}, "hash": "89ee532b7d56a10764bc114f8dd5f5ed9b8e09da8c99c87e082d0e39eca7e8b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7249, "end_char_idx": 9037, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Trim the start of the original audio file
trimmed_audio = trim_start(earnings_call_filepath)

```


```python
trimmed_audio, trimmed_filename = trim_start(earnings_call_filepath)

```

Our fake earnings report audio file is fairly short in length, so we'll adjust the segments accordingly. Keep in mind you can adjust the segment length as you need.


```python
# Segment audio
trimmed_audio = AudioSegment.from_wav(trimmed_filename)  # Load the trimmed audio file

one_minute = 1 * 60 * 1000  # Duration for each segment (in milliseconds)

start_time = 0  # Start time for the first segment

i = 0  # Index for naming the segmented files

output_dir_trimmed = "trimmed_earnings_directory"  # Output directory for the segmented files

if not os.path.isdir(output_dir_trimmed):  # Create the output directory if it does not exist
    os.makedirs(output_dir_trimmed)

while start_time < len(trimmed_audio):  # Loop over the trimmed audio file
    segment = trimmed_audio[start_time:start_time + one_minute]  # Extract a segment
    segment.export(os.path.join(output_dir_trimmed, f"trimmed_{i:02d}.wav"), format="wav")  # Save the segment
    start_time += one_minute  # Update the start time for the next segment
    i += 1  # Increment the index for naming the next file

```


```python
# Get list of trimmed and segmented audio files and sort them numerically
audio_files = sorted(
    (f for f in os.listdir(output_dir_trimmed) if f.endswith(".wav")),
    key=lambda f: int(''.join(filter(str.isdigit, f)))
)

```


```python
# Use a loop to apply the transcribe function to all audio files
transcriptions = [transcribe_audio(file, output_dir_trimmed) for file in audio_files]

```


```python
# Concatenate the transcriptions
full_transcript = ' '.join(transcriptions)
```


key: _node_content
string_value:
{"id_": "c707fd65-16fa-4add-89d7-84da12469f5e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b906d5a1-3a54-4b25-b149-e819d43a0a1f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4b1cba78d8aa7739b0f179e95de7686909f630c919bcff5dde85720e3fd670ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b54256f2-b034-4631-9d39-e4cb27a08408", "node_type": "1", "metadata": {}, "hash": "72c5e5e2d75b959ff6b4aca1db084e20fe4e3fba4b65ec1826257384dc8a8d30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7613, "end_char_idx": 11303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Segment audio
trimmed_audio = AudioSegment.from_wav(trimmed_filename)  # Load the trimmed audio file

one_minute = 1 * 60 * 1000  # Duration for each segment (in milliseconds)

start_time = 0  # Start time for the first segment

i = 0  # Index for naming the segmented files

output_dir_trimmed = "trimmed_earnings_directory"  # Output directory for the segmented files

if not os.path.isdir(output_dir_trimmed):  # Create the output directory if it does not exist
    os.makedirs(output_dir_trimmed)

while start_time < len(trimmed_audio):  # Loop over the trimmed audio file
    segment = trimmed_audio[start_time:start_time + one_minute]  # Extract a segment
    segment.export(os.path.join(output_dir_trimmed, f"trimmed_{i:02d}.wav"), format="wav")  # Save the segment
    start_time += one_minute  # Update the start time for the next segment
    i += 1  # Increment the index for naming the next file

```


```python
# Get list of trimmed and segmented audio files and sort them numerically
audio_files = sorted(
    (f for f in os.listdir(output_dir_trimmed) if f.endswith(".wav")),
    key=lambda f: int(''.join(filter(str.isdigit, f)))
)

```


```python
# Use a loop to apply the transcribe function to all audio files
transcriptions = [transcribe_audio(file, output_dir_trimmed) for file in audio_files]

```


```python
# Concatenate the transcriptions
full_transcript = ' '.join(transcriptions)
```


```python
print(full_transcript)
```

    Good afternoon, everyone. And welcome to FinTech Plus Sync's second quarter 2023 earnings call. I'm John Doe, CEO of FinTech Plus. We've had a stellar Q2 with a revenue of 125 million, a 25% increase year over year. Our gross profit margin stands at a solid 58%, due in part to cost efficiencies gained from our scalable business model. Our EBITDA has surged to 37.5 million, translating to a remarkable 30% EBITDA margin. Our net income for the quarter rose to 16 million, which is a noteworthy increase from 10 million in Q2 2022. Our total addressable market has grown substantially thanks to the expansion of our high yield savings product line and the new RoboAdvisor platform. We've been diversifying our asset-backed securities portfolio, investing heavily in collateralized. debt obligations, and residential mortgage-backed securities. We've also invested $25 million in AAA rated corporate bonds, enhancing our risk adjusted returns. As for our balance sheet, total assets reached $1.5 billion with total liabilities at $900 million, leaving us with a solid equity base of $600 million. Our debt-to-equity ratio stands at 1.5, a healthy figure considering our expansionary phase. We continue to see substantial organic user growth, with customer acquisition cost dropping by 15% and lifetime value growing by 25%. Our LTVCAC ratio is at an impressive 3.5%. In terms of risk management, we have a value-at-risk model in place with a 99%... confidence level indicating that our maximum loss will not exceed 5 million in the next trading day. We've adopted a conservative approach to managing our leverage and have a healthy tier one capital ratio of 12.5%. Our forecast for the coming quarter is positive. We expect revenue to be around 135 million and 8% quarter over quarter growth driven primarily by our cutting edge blockchain solutions and AI driven predictive analytics. We're also excited about the upcoming IPO of our FinTech subsidiary Pay Plus, which we expect to raise 200 million, significantly bolstering our liquidity and paving the way for aggressive growth strategies. We thank our shareholders for their continued faith in us and we look forward to an even more successful Q3. Thank you so much.


key: _node_content
string_value:
{"id_": "b54256f2-b034-4631-9d39-e4cb27a08408", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c707fd65-16fa-4add-89d7-84da12469f5e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "49b0ef804ba03ee2e1f7d9f64a4b2f1e0c7709f1e9bcc62217b0cb97ef1dd57f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1356c26e-cf35-4be3-98c1-91aa94acee1c", "node_type": "1", "metadata": {}, "hash": "a09ab78b36bb4244b7e67208f56c917b7256e350437cf29fc68c3ac79326f7b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11307, "end_char_idx": 13934, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Remove non-ascii characters from the transcript
ascii_transcript = remove_non_ascii(full_transcript)
```


```python
print(ascii_transcript)
```

    Good afternoon, everyone. And welcome to FinTech Plus Sync's second quarter 2023 earnings call. I'm John Doe, CEO of FinTech Plus. We've had a stellar Q2 with a revenue of 125 million, a 25% increase year over year. Our gross profit margin stands at a solid 58%, due in part to cost efficiencies gained from our scalable business model. Our EBITDA has surged to 37.5 million, translating to a remarkable 30% EBITDA margin. Our net income for the quarter rose to 16 million, which is a noteworthy increase from 10 million in Q2 2022. Our total addressable market has grown substantially thanks to the expansion of our high yield savings product line and the new RoboAdvisor platform. We've been diversifying our asset-backed securities portfolio, investing heavily in collateralized. debt obligations, and residential mortgage-backed securities. We've also invested $25 million in AAA rated corporate bonds, enhancing our risk adjusted returns. As for our balance sheet, total assets reached $1.5 billion with total liabilities at $900 million, leaving us with a solid equity base of $600 million. Our debt-to-equity ratio stands at 1.5, a healthy figure considering our expansionary phase. We continue to see substantial organic user growth, with customer acquisition cost dropping by 15% and lifetime value growing by 25%. Our LTVCAC ratio is at an impressive 3.5%. In terms of risk management, we have a value-at-risk model in place with a 99%... confidence level indicating that our maximum loss will not exceed 5 million in the next trading day. We've adopted a conservative approach to managing our leverage and have a healthy tier one capital ratio of 12.5%. Our forecast for the coming quarter is positive. We expect revenue to be around 135 million and 8% quarter over quarter growth driven primarily by our cutting edge blockchain solutions and AI driven predictive analytics. We're also excited about the upcoming IPO of our FinTech subsidiary Pay Plus, which we expect to raise 200 million, significantly bolstering our liquidity and paving the way for aggressive growth strategies. We thank our shareholders for their continued faith in us and we look forward to an even more successful Q3. Thank you so much.



```python
# Use punctuation assistant function
response = punctuation_assistant(ascii_transcript)
```


```python
# Extract the punctuated transcript from the model's response
punctuated_transcript = response.choices[0].message.content

```


key: _node_content
string_value:
{"id_": "1356c26e-cf35-4be3-98c1-91aa94acee1c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b54256f2-b034-4631-9d39-e4cb27a08408", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0d3f79b38f777b8e52ddb7173c6801ea13a0cec662360da14856f5f0e160e94b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e99aebd7-f30c-471a-933a-c263e5deca66", "node_type": "1", "metadata": {}, "hash": "7a40d3431e7ccc2e456c6789aa456a094be82379c6df6fd6e6fc3e28e4597763", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13694, "end_char_idx": 16440, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Use punctuation assistant function
response = punctuation_assistant(ascii_transcript)
```


```python
# Extract the punctuated transcript from the model's response
punctuated_transcript = response.choices[0].message.content

```


```python
print(punctuated_transcript)
```

    Good afternoon, everyone. And welcome to FinTech Plus Sync's second quarter 2023 earnings call. I'm John Doe, CEO of FinTech Plus. We've had a stellar Q2 with a revenue of $125 million, a 25% increase year over year. Our gross profit margin stands at a solid 58%, due in part to cost efficiencies gained from our scalable business model. Our EBITDA has surged to $37.5 million, translating to a remarkable 30% EBITDA margin. Our net income for the quarter rose to $16 million, which is a noteworthy increase from $10 million in Q2 2022. Our total addressable market has grown substantially thanks to the expansion of our high yield savings product line and the new RoboAdvisor platform. We've been diversifying our asset-backed securities portfolio, investing heavily in collateralized debt obligations, and residential mortgage-backed securities. We've also invested $25 million in AAA rated corporate bonds, enhancing our risk-adjusted returns. As for our balance sheet, total assets reached $1.5 billion with total liabilities at $900 million, leaving us with a solid equity base of $600 million. Our debt-to-equity ratio stands at 1.5, a healthy figure considering our expansionary phase. We continue to see substantial organic user growth, with customer acquisition cost dropping by 15% and lifetime value growing by 25%. Our LTVCAC ratio is at an impressive 3.5%. In terms of risk management, we have a value-at-risk model in place with a 99% confidence level indicating that our maximum loss will not exceed $5 million in the next trading day. We've adopted a conservative approach to managing our leverage and have a healthy tier one capital ratio of 12.5%. Our forecast for the coming quarter is positive. We expect revenue to be around $135 million and 8% quarter over quarter growth driven primarily by our cutting-edge blockchain solutions and AI-driven predictive analytics. We're also excited about the upcoming IPO of our FinTech subsidiary Pay Plus, which we expect to raise $200 million, significantly bolstering our liquidity and paving the way for aggressive growth strategies. We thank our shareholders for their continued faith in us and we look forward to an even more successful Q3. Thank you so much.



```python
# Use product assistant function
response = product_assistant(punctuated_transcript)

```


```python
# Extract the final transcript from the model's response
final_transcript = response.choices[0].message.content
```


key: _node_content
string_value:
{"id_": "e99aebd7-f30c-471a-933a-c263e5deca66", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "df12db80-b094-4bd6-98f1-c5a6cce71abe", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc43f25e4bad646dcfd48f7eba342e2e601e53c1f3ce2232fc15f7992037e6fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1356c26e-cf35-4be3-98c1-91aa94acee1c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_processing_guide.ipynb", "title": "Enhancing Whisper transcriptions: pre- & post-processing techniques", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "569c2ed5cbf82bc6eac8a0efacafb65a100ee39ba8490af70b37a18cc7e8e677", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16213, "end_char_idx": 19472, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Use product assistant function
response = product_assistant(punctuated_transcript)

```


```python
# Extract the final transcript from the model's response
final_transcript = response.choices[0].message.content
```


```python
print(final_transcript)
```

    Good afternoon, everyone. And welcome to FinTech Plus Sync's second quarter 2023 earnings call. I'm John Doe, CEO of FinTech Plus. We've had a stellar second quarter (Q2) with a revenue of $125 million, a 25% increase year over year. Our gross profit margin stands at a solid 58%, due in part to cost efficiencies gained from our scalable business model. Our Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA) has surged to $37.5 million, translating to a remarkable 30% EBITDA margin. Our net income for the quarter rose to $16 million, which is a noteworthy increase from $10 million in second quarter (Q2) 2022. Our total addressable market has grown substantially thanks to the expansion of our high yield savings product line and the new RoboAdvisor platform. We've been diversifying our asset-backed securities portfolio, investing heavily in Collateralized Debt Obligations (CDOs), and Residential Mortgage-Backed Securities (RMBS). We've also invested $25 million in AAA rated corporate bonds, enhancing our risk-adjusted returns. As for our balance sheet, total assets reached $1.5 billion with total liabilities at $900 million, leaving us with a solid equity base of $600 million. Our Debt-to-Equity (D/E) ratio stands at 1.5, a healthy figure considering our expansionary phase. We continue to see substantial organic user growth, with Customer Acquisition Cost (CAC) dropping by 15% and Lifetime Value (LTV) growing by 25%. Our LTV to CAC (LTVCAC) ratio is at an impressive 3.5%. In terms of risk management, we have a Value at Risk (VaR) model in place with a 99% confidence level indicating that our maximum loss will not exceed $5 million in the next trading day. We've adopted a conservative approach to managing our leverage and have a healthy Tier 1 Capital ratio of 12.5%. Our forecast for the coming quarter is positive. We expect revenue to be around $135 million and 8% quarter over quarter growth driven primarily by our cutting-edge blockchain solutions and AI-driven predictive analytics. We're also excited about the upcoming Initial Public Offering (IPO) of our FinTech subsidiary Pay Plus, which we expect to raise $200 million, significantly bolstering our liquidity and paving the way for aggressive growth strategies. We thank our shareholders for their continued faith in us and we look forward to an even more successful third quarter (Q3). Thank you so much.
    
    Words Changed:
    1. Q2 -> second quarter (Q2)
    2. EBITDA -> Earnings Before Interest, Taxes, Depreciation, and Amortization (EBITDA)
    3. Q2 2022 -> second quarter (Q2) 2022
    4. CDOs -> Collateralized Debt Obligations (CDOs)
    5. RMBS -> Residential Mortgage-Backed Securities (RMBS)
    6. D/E -> Debt-to-Equity (D/E)
    7. CAC -> Customer Acquisition Cost (CAC)
    8. LTV -> Lifetime Value (LTV)
    9. LTVCAC -> LTV to CAC (LTVCAC)
    10. VaR -> Value at Risk (VaR)
    11. IPO -> Initial Public Offering (IPO)
    12. Q3 -> third quarter (Q3)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb


key: _node_content
string_value:
{"id_": "2e8eca75-b6d7-4929-b831-95ae2dcd873d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3888f48-d5fc-495b-b446-da76f1f01cd9", "node_type": "1", "metadata": {}, "hash": "c4254d8c62443e86b16503c909db9f2a9579fa2e50c7ccc6501a14d591220afa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question answering using a search API and re-ranking

Searching for relevant information can sometimes feel like looking for a needle in a haystack, but don’t despair, GPTs can actually do a lot of this work for us. In this guide we explore a way to augment existing search systems with various AI techniques, helping us sift through the noise.

Two ways of retrieving information for GPT are:

1. **Mimicking Human Browsing:** [GPT triggers a search](https://openai.com/blog/chatgpt-plugins#browsing), evaluates the results, and modifies the search query if necessary. It can also follow up on specific search results to form a chain of thought, much like a human user would do.
2. **Retrieval with Embeddings:** Calculate [embeddings](https://platform.openai.com/docs/guides/embeddings) for your content and a user query, and then [retrieve the content](Question_answering_using_embeddings.ipynb) most related as measured by cosine similarity. This technique is [used heavily](https://blog.google/products/search/search-language-understanding-bert/) by search engines like Google.

These approaches are both promising, but each has their shortcomings: the first one can be slow due to its iterative nature and the second one requires embedding your entire knowledge base in advance, continuously embedding new content and maintaining a vector database.

By combining these approaches, and drawing inspiration from [re-ranking](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) methods, we identify an approach that sits in the middle. **This approach can be implemented on top of any existing search system, like the Slack search API, or an internal ElasticSearch instance with private data**. Here’s how it works:

![search_augmented_by_query_generation_and_embeddings_reranking.png](../images/search_rerank_answer.png)

**Step 1: Search**

1.  User asks a question.
2.  GPT generates a list of potential queries.
3.  Search queries are executed in parallel.

**Step 2: Re-rank**

1.  Embeddings for each result are used to calculate semantic similarity to a generated hypothetical ideal answer to the user question.
2.  Results are ranked and filtered based on this similarity metric.

**Step 3: Answer**

1.  Given the top search results, the model generates an answer to the user’s question, including references and links.

This hybrid approach offers relatively low latency and can be integrated into any existing search endpoint, without requiring the upkeep of a vector database. Let's dive into it! We will use the [News API](https://newsapi.org/) as an example domain to search over.

## Setup

In addition to your `OPENAI_API_KEY`, you'll have to include a `NEWS_API_KEY` in your environment. You can get an API key [here](https://newsapi.org/).



```python
%%capture
%env NEWS_API_KEY = YOUR_NEWS_API_KEY

```


key: _node_content
string_value:
{"id_": "c3888f48-d5fc-495b-b446-da76f1f01cd9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e8eca75-b6d7-4929-b831-95ae2dcd873d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "56bc523ca19b457691a16fd1be45bcb82be3dab3a4a0b4513c1c2b6ec2814e15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c11fd1d4-df32-4a3f-9cea-6149b70abfad", "node_type": "1", "metadata": {}, "hash": "ed9d4b732ef247f18cba6b0c12366e7e1a84df9d7dc3bb4c031147fceb797ccf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2787, "end_char_idx": 5980, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
%%capture
%env NEWS_API_KEY = YOUR_NEWS_API_KEY

```


```python
# Dependencies
from datetime import date, timedelta  # date handling for fetching recent news
from IPython import display  # for pretty printing
import json  # for parsing the JSON api responses and model outputs
from numpy import dot  # for cosine similarity
from openai import OpenAI
import os  # for loading environment variables
import requests  # for making the API requests
from tqdm.notebook import tqdm  # for printing progress bars

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

# Load environment variables
news_api_key = os.getenv("NEWS_API_KEY")

GPT_MODEL = "gpt-3.5-turbo"


# Helper functions
def json_gpt(input: str):
    completion = client.chat.completions.create(model=GPT_MODEL,
    messages=[
        {"role": "system", "content": "Output only valid JSON"},
        {"role": "user", "content": input},
    ],
    temperature=0.5)

    text = completion.choices[0].message.content
    parsed = json.loads(text)

    return parsed


def embeddings(input: list[str]) -> list[list[str]]:
    response = client.embeddings.create(model="text-embedding-3-small", input=input)
    return [data.embedding for data in response.data]
```

## 1. Search

It all starts with a user question.



```python
# User asks a question
USER_QUESTION = "Who won the NBA championship? And who was the MVP? Tell me a bit about the last game."
```

Now, in order to be as exhaustive as possible, we use the model to generate a list of diverse queries based on this question.



```python
QUERIES_INPUT = f"""
You have access to a search API that returns recent news articles.
Generate an array of search queries that are relevant to this question.
Use a variation of related keywords for the queries, trying to be as general as possible.
Include as many queries as you can think of, including and excluding terms.
For example, include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2'].
Be creative. The more queries you include, the more likely you are to find relevant results.

User question: {USER_QUESTION}

Format: {{"queries": ["query_1", "query_2", "query_3"]}}
"""

queries = json_gpt(QUERIES_INPUT)["queries"]

# Let's include the original question as well for good measure
queries.append(USER_QUESTION)

queries
```




    ['NBA championship winner',
     'MVP of NBA championship',
     'Last game of NBA championship',
     'NBA finals winner',
     'Most valuable player of NBA championship',
     'Finals game of NBA',
     'Who won the NBA finals',
     'NBA championship game summary',
     'NBA finals MVP',
     'Champion of NBA playoffs',
     'NBA finals last game highlights',
     'NBA championship series result',
     'NBA finals game score',
     'NBA finals game recap',
     'NBA champion team and player',
     'NBA finals statistics',
     'NBA championship final score',
     'NBA finals best player',
     'NBA playoffs champion and MVP',
     'NBA finals game analysis',
     'Who won the NBA championship? And who was the MVP? Tell me a bit about the last game.']



The queries look good, so let's run the searches.


key: _node_content
string_value:
{"id_": "c11fd1d4-df32-4a3f-9cea-6149b70abfad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3888f48-d5fc-495b-b446-da76f1f01cd9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "be84bdd4e7fdd3fc40c69cbdff1bcc766e94841454c53c6594e1f64b2f7d1db5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9a7a288e-83b5-44e4-a04d-a31711b79242", "node_type": "1", "metadata": {}, "hash": "aeb0caee2ade7eac44215e7ed07742a4daf74ee655d796dee1294b507b632be2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4395, "end_char_idx": 7202, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
QUERIES_INPUT = f"""
You have access to a search API that returns recent news articles.
Generate an array of search queries that are relevant to this question.
Use a variation of related keywords for the queries, trying to be as general as possible.
Include as many queries as you can think of, including and excluding terms.
For example, include queries like ['keyword_1 keyword_2', 'keyword_1', 'keyword_2'].
Be creative. The more queries you include, the more likely you are to find relevant results.

User question: {USER_QUESTION}

Format: {{"queries": ["query_1", "query_2", "query_3"]}}
"""

queries = json_gpt(QUERIES_INPUT)["queries"]

# Let's include the original question as well for good measure
queries.append(USER_QUESTION)

queries
```




    ['NBA championship winner',
     'MVP of NBA championship',
     'Last game of NBA championship',
     'NBA finals winner',
     'Most valuable player of NBA championship',
     'Finals game of NBA',
     'Who won the NBA finals',
     'NBA championship game summary',
     'NBA finals MVP',
     'Champion of NBA playoffs',
     'NBA finals last game highlights',
     'NBA championship series result',
     'NBA finals game score',
     'NBA finals game recap',
     'NBA champion team and player',
     'NBA finals statistics',
     'NBA championship final score',
     'NBA finals best player',
     'NBA playoffs champion and MVP',
     'NBA finals game analysis',
     'Who won the NBA championship? And who was the MVP? Tell me a bit about the last game.']



The queries look good, so let's run the searches.



```python
def search_news(
    query: str,
    news_api_key: str = news_api_key,
    num_articles: int = 50,
    from_datetime: str = "2023-06-01",  # the 2023 NBA finals were played in June 2023
    to_datetime: str = "2023-06-30",
) -> dict:
    response = requests.get(
        "https://newsapi.org/v2/everything",
        params={
            "q": query,
            "apiKey": news_api_key,
            "pageSize": num_articles,
            "sortBy": "relevancy",
            "from": from_datetime,
            "to": to_datetime,
        },
    )

    return response.json()


articles = []

for query in tqdm(queries):
    result = search_news(query)
    if result["status"] == "ok":
        articles = articles + result["articles"]
    else:
        raise Exception(result["message"])

# remove duplicates
articles = list({article["url"]: article for article in articles}.values())

print("Total number of articles:", len(articles))
print("Top 5 articles of query 1:", "\n")

for article in articles[0:5]:
    print("Title:", article["title"])
    print("Description:", article["description"])
    print("Content:", article["content"][0:100] + "...")
    print()

```


      0%|          | 0/21 [00:00<?, ?it/s]


key: _node_content
string_value:
{"id_": "9a7a288e-83b5-44e4-a04d-a31711b79242", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c11fd1d4-df32-4a3f-9cea-6149b70abfad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "95dda123d329dc8963a7b011c9ac48603182a2033f96d1bacf44a48be0464c3b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b5b8797-4bbf-43c0-8588-76d72a8a3dde", "node_type": "1", "metadata": {}, "hash": "4925777faecc5576431ca759e371dfac1edc1e951f5c6a42f16d812accbc3b01", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5931, "end_char_idx": 9828, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The queries look good, so let's run the searches.



```python
def search_news(
    query: str,
    news_api_key: str = news_api_key,
    num_articles: int = 50,
    from_datetime: str = "2023-06-01",  # the 2023 NBA finals were played in June 2023
    to_datetime: str = "2023-06-30",
) -> dict:
    response = requests.get(
        "https://newsapi.org/v2/everything",
        params={
            "q": query,
            "apiKey": news_api_key,
            "pageSize": num_articles,
            "sortBy": "relevancy",
            "from": from_datetime,
            "to": to_datetime,
        },
    )

    return response.json()


articles = []

for query in tqdm(queries):
    result = search_news(query)
    if result["status"] == "ok":
        articles = articles + result["articles"]
    else:
        raise Exception(result["message"])

# remove duplicates
articles = list({article["url"]: article for article in articles}.values())

print("Total number of articles:", len(articles))
print("Top 5 articles of query 1:", "\n")

for article in articles[0:5]:
    print("Title:", article["title"])
    print("Description:", article["description"])
    print("Content:", article["content"][0:100] + "...")
    print()

```


      0%|          | 0/21 [00:00<?, ?it/s]


    Total number of articles: 554
    Top 5 articles of query 1: 
    
    Title: Nascar takes on Le Mans as LeBron James gets centenary race under way
    Description: <ul><li>Nascar has presence at iconic race for first time since 1976</li><li>NBA superstar LeBron James waves flag as honorary starter</li></ul>The crowd chanted “U-S-A! U-S-A!” as Nascar driver lineup for the 24 Hours of Le Mans passed through the city cente…
    Content: The crowd chanted U-S-A! U-S-A! as Nascar driver lineup for the 24 Hours of Le Mans passed through t...
    
    Title: NBA finals predictions: Nuggets or Heat? Our writers share their picks
    Description: Denver or Miami? Our contributors pick the winner, key players and dark horses before the NBA’s grand finale tips offA lot has been made of the importance of a balanced roster with continuity, but, somehow, still not enough. The Nuggets are the prime example …
    Content: The Nuggets are here because 
    A lot has been made of the importance of a balanced roster with conti...
    
    Title: Unboxing: Michelob ULTRA and Artist Futura Enshrine the NBA Championship In Custom Hand-Painted Bottles
    Description: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will gather to watch the showdown between the Denver Nuggets and Miami Heat. The beermaker teamed up with artist Futura to remix its newly-designed 2023 Champ Bottle…
    Content: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will g...
    
    Title: Futura and Michelob ULTRA Toast to the NBA Finals With Abstract Artwork Crafted From the Brand’s 2023 Limited-Edition Championship Bottles
    Description: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beermaker is back with its celebratory NBA Champ Bottles. This year, the self-proclaimed MVP of joy is dropping a limited-edition bottle made in collaboration with a…
    Content: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beerma...
    
    Title: Signed and Delivered, Futura and Michelob ULTRA Will Gift Hand-Painted Bottles to This Year’s NBA Championship Team
    Description: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basketball lovers and sports fans around the globe as the NBA 2022-2023 season comes to a nail-biting close. In collaboration with artist Futura, Michelob ULTRA will…
    Content: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basket...


key: _node_content
string_value:
{"id_": "7b5b8797-4bbf-43c0-8588-76d72a8a3dde", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9a7a288e-83b5-44e4-a04d-a31711b79242", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc92a8ac8cc3f2ade2d0ff2bcda308017f9f4142da0520beed75e1b421c8a9c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dacd88b8-4482-456d-b202-cf59df0b09a6", "node_type": "1", "metadata": {}, "hash": "943746c439027f798ce77ace1d780f4e946f3f4546acfa84311cbaf7d3eddc71", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9836, "end_char_idx": 12345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As we can see, oftentimes, the search queries will return a large number of results, many of which are not relevant to the original question asked by the user. In order to improve the quality of the final answer, we use embeddings to re-rank and filter the results.

## 2. Re-rank

Drawing inspiration from [HyDE (Gao et al.)](https://arxiv.org/abs/2212.10496), we first generate a hypothetical ideal answer to rerank our compare our results against. This helps prioritize results that look like good answers, rather than those similar to our question. Here’s the prompt we use to generate our hypothetical answer.



```python
HA_INPUT = f"""
Generate a hypothetical answer to the user's question. This answer will be used to rank search results. 
Pretend you have all the information you need to answer, but don't use any actual facts. Instead, use placeholders
like NAME did something, or NAME said something at PLACE. 

User question: {USER_QUESTION}

Format: {{"hypotheticalAnswer": "hypothetical answer text"}}
"""

hypothetical_answer = json_gpt(HA_INPUT)["hypotheticalAnswer"]

hypothetical_answer

```




    'The NBA championship was won by TEAM NAME. The MVP was awarded to PLAYER NAME. The last game was held at STADIUM NAME, where both teams played with great energy and enthusiasm. It was a close game, but in the end, TEAM NAME emerged victorious.'



Now, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.



```python
hypothetical_answer_embedding = embeddings(hypothetical_answer)[0]
article_embeddings = embeddings(
    [
        f"{article['title']} {article['description']} {article['content'][0:100]}"
        for article in articles
    ]
)

# Calculate cosine similarity
cosine_similarities = []
for article_embedding in article_embeddings:
    cosine_similarities.append(dot(hypothetical_answer_embedding, article_embedding))

cosine_similarities[0:10]

```




    [0.7854456526852069,
     0.8086023500072106,
     0.8002998147018501,
     0.7961229569526956,
     0.798354506673743,
     0.758216458795653,
     0.7753754083127359,
     0.7494958338411927,
     0.804733946801739,
     0.8405965885235218]



Finally, we use these similarity scores to sort and filter the results.


key: _node_content
string_value:
{"id_": "dacd88b8-4482-456d-b202-cf59df0b09a6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b5b8797-4bbf-43c0-8588-76d72a8a3dde", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "aa5960b39569e3dc8cd819d33d63f8ba93f109b8ad6b50aead983d3f5058c697", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1eb60d61-603a-4c5a-9f04-f07981961313", "node_type": "1", "metadata": {}, "hash": "8928b7b99b2e7d118441c4164cf190081aa3eabe0c5e0f0478aa153ed8c50f87", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10955, "end_char_idx": 14906, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'The NBA championship was won by TEAM NAME. The MVP was awarded to PLAYER NAME. The last game was held at STADIUM NAME, where both teams played with great energy and enthusiasm. It was a close game, but in the end, TEAM NAME emerged victorious.'



Now, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.



```python
hypothetical_answer_embedding = embeddings(hypothetical_answer)[0]
article_embeddings = embeddings(
    [
        f"{article['title']} {article['description']} {article['content'][0:100]}"
        for article in articles
    ]
)

# Calculate cosine similarity
cosine_similarities = []
for article_embedding in article_embeddings:
    cosine_similarities.append(dot(hypothetical_answer_embedding, article_embedding))

cosine_similarities[0:10]

```




    [0.7854456526852069,
     0.8086023500072106,
     0.8002998147018501,
     0.7961229569526956,
     0.798354506673743,
     0.758216458795653,
     0.7753754083127359,
     0.7494958338411927,
     0.804733946801739,
     0.8405965885235218]



Finally, we use these similarity scores to sort and filter the results.



```python
scored_articles = zip(articles, cosine_similarities)

# Sort articles by cosine similarity
sorted_articles = sorted(scored_articles, key=lambda x: x[1], reverse=True)

# Print top 5 articles
print("Top 5 articles:", "\n")

for article, score in sorted_articles[0:5]:
    print("Title:", article["title"])
    print("Description:", article["description"])
    print("Content:", article["content"][0:100] + "...")
    print("Score:", score)
    print()

```

    Top 5 articles: 
    
    Title: NBA Finals: Denver Nuggets beat Miami Hea, lift thier first-ever NBA title
    Description: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the NBA Final held on Tuesday at the Ball Arena in Denver
    Content: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the ...
    Score: 0.8445817523602124
    
    Title: Photos: Denver Nuggets celebrate their first NBA title
    Description: The Nuggets capped off an impressive postseason by beating the Miami Heat in the NBA Finals.
    Content: Thousands of supporters watched along the streets of Denver, Colorado as the US National Basketball ...
    Score: 0.842070667753606
    
    Title: Denver Nuggets win first NBA championship title in Game 5 victory over Miami Heat
    Description: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ball Arena in Denver to take Game 5 of the NBA Finals.
    Content: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ba...
    Score: 0.8409346078172385
    
    Title: Denver Nuggets Capture Their First NBA Championship Behind Unbreakable Chemistry
    Description: After 47 years of waiting, the Denver Nuggets are NBA champions. Led by Nikola Jokic and Jamal Murray, they reached the mountain top by staying true to themselves.
    Content: DENVER, CO - JUNE 12: Jamal Murray (27) of the Denver Nuggets celebrates as he leaves the court ... ...
    Score: 0.8405965885235218
    
    Title: NBA Finals: Nikola Jokic, Denver Nuggets survive Miami Heat to secure franchise's first NBA championship
    Description: In a rock-fight of a Game 5, the Denver Nuggets reached the NBA mountaintop from the foothills of the Rockies, winning their first-ever championship and setting Nikola Jokic's legacy as an all-timer in stone.
    Content: DENVER, COLORADO - JUNE 12: Jamal Murray #27 of the Denver Nuggets reacts during the fourth quarter ...
    Score: 0.8389716330890262


key: _node_content
string_value:
{"id_": "1eb60d61-603a-4c5a-9f04-f07981961313", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dacd88b8-4482-456d-b202-cf59df0b09a6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d68ca5d96da23fd657418ebdc2a3e16a37aed47a0705f3ece76fdc87e0809d40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14914, "end_char_idx": 17018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Awesome! These results look a lot more relevant to our original query. Now, let's use the top 5 results to generate a final answer.

## 3. Answer



```python
formatted_top_results = [
    {
        "title": article["title"],
        "description": article["description"],
        "url": article["url"],
    }
    for article, _score in sorted_articles[0:5]
]

ANSWER_INPUT = f"""
Generate an answer to the user's question based on the given search results. 
TOP_RESULTS: {formatted_top_results}
USER_QUESTION: {USER_QUESTION}

Include as much information as possible in the answer. Reference the relevant search result urls as markdown links.
"""

completion = client.chat.completions.create(
    model=GPT_MODEL,
    messages=[{"role": "user", "content": ANSWER_INPUT}],
    temperature=0.5,
    stream=True,
)

text = ""
for chunk in completion:
    text += chunk.choices[0].delta.content
    display.clear_output(wait=True)
    display.display(display.Markdown(text))
```


The Denver Nuggets won their first-ever NBA championship by defeating the Miami Heat 94-89 in Game 5 of the NBA Finals held on Tuesday at the Ball Arena in Denver, according to this [Business Standard article](https://www.business-standard.com/sports/other-sports-news/nba-finals-denver-nuggets-beat-miami-hea-lift-thier-first-ever-nba-title-123061300285_1.html). Nikola Jokic, the Nuggets' center, was named the NBA Finals MVP. In a rock-fight of a Game 5, the Nuggets reached the NBA mountaintop, securing their franchise's first NBA championship and setting Nikola Jokic's legacy as an all-timer in stone, according to this [Yahoo Sports article](https://sports.yahoo.com/nba-finals-nikola-jokic-denver-nuggets-survive-miami-heat-to-secure-franchises-first-nba-championship-030321214.html). For more information and photos of the Nuggets' celebration, check out this [Al Jazeera article](https://www.aljazeera.com/gallery/2023/6/15/photos-denver-nuggets-celebrate-their-first-nba-title) and this [CNN article](https://www.cnn.com/2023/06/12/sport/denver-nuggets-nba-championship-spt-intl?cid=external-feeds_iluminar_yahoo).


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb


key: _node_content
string_value:
{"id_": "17236226-3963-416a-93d1-5f65aade71c6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9daa6650-83d9-469d-a903-178e3bd488ae", "node_type": "1", "metadata": {}, "hash": "16a59da98549bc0110606c7cd53af1102ff09372c7b472dfbb3e335e33c89672", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3275, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Search reranking with cross-encoders

This notebook takes you through examples of using a cross-encoder to re-rank search results.

This is a common use case with our customers, where you've implemented semantic search using embeddings (produced using a [bi-encoder](https://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieval-bi-encoder)) but the results are not as accurate as your use case requires. A possible cause is that there is some business rule you can use to rerank the documents such as how recent or how popular a document is. 

However, often there are subtle domain-specific rules that help determine relevancy, and this is where a cross-encoder can be useful. Cross-encoders are more accurate than bi-encoders but they don't scale well, so using them to re-order a shortened list returned by semantic search is the ideal use case.

### Example

Consider a search task with D documents and Q queries.

The brute force approach of computing every pairwise relevance is expensive; its cost scales as ```D * Q```. This is known as **cross-encoding**.

A faster approach is **embeddings-based search**, in which an embedding is computed once for each document and query, and then re-used multiple times to cheaply compute pairwise relevance. Because embeddings are only computed once, its cost scales as ```D + Q```. This is known as **bi-encoding**.

Although embeddings-based search is faster, the quality can be worse. To get the best of both, one common approach is to use embeddings (or another bi-encoder) to cheaply identify top candidates, and then use GPT (or another cross-encoder) to expensively re-rank those top candidates. The cost of this hybrid approach scales as ```(D + Q) * cost of embedding + (N * Q) * cost of re-ranking```, where ```N``` is the number of candidates re-ranked.

### Walkthrough

To illustrate this approach we'll use ```text-davinci-003``` with ```logprobs``` enabled to build a GPT-powered cross-encoder. Our GPT models have strong general language understanding, which when tuned with some few-shot examples can provide a simple and effective cross-encoding option.

This notebook drew on this great [article](https://weaviate.io/blog/cross-encoders-as-reranker) by Weaviate, and this [excellent explanation](https://www.sbert.net/examples/applications/cross-encoder/README.html) of bi-encoders vs. cross-encoders from Sentence Transformers.


```python
!pip install openai
!pip install arxiv
!pip install tenacity
!pip install pandas
!pip install tiktoken
```


```python
import arxiv
from math import exp
import openai
import os
import pandas as pd
from tenacity import retry, wait_random_exponential, stop_after_attempt
import tiktoken

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

OPENAI_MODEL = "gpt-4"
```

## Search

We'll use the arXiv search service for this example, but this step could be performed by any search service you have. The key item to consider is over-fetching slightly to capture all the potentially relevant documents, before re-sorting them.



```python
query = "how do bi-encoders work for sentence embeddings"
search = arxiv.Search(
    query=query, max_results=20, sort_by=arxiv.SortCriterion.Relevance
)
```


key: _node_content
string_value:
{"id_": "9daa6650-83d9-469d-a903-178e3bd488ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17236226-3963-416a-93d1-5f65aade71c6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6888283326d02001635a0d0efefe7d765f74a37bb7108add97355b3bd22918bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3def4ee-a9fa-495e-960d-4900d0114fd0", "node_type": "1", "metadata": {}, "hash": "3fdb09420e94baba44c98dfab1fd368e4cd8dce2bf4c9e74fce31e2f27fcbb49", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2420, "end_char_idx": 5391, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install openai
!pip install arxiv
!pip install tenacity
!pip install pandas
!pip install tiktoken
```


```python
import arxiv
from math import exp
import openai
import os
import pandas as pd
from tenacity import retry, wait_random_exponential, stop_after_attempt
import tiktoken

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

OPENAI_MODEL = "gpt-4"
```

## Search

We'll use the arXiv search service for this example, but this step could be performed by any search service you have. The key item to consider is over-fetching slightly to capture all the potentially relevant documents, before re-sorting them.



```python
query = "how do bi-encoders work for sentence embeddings"
search = arxiv.Search(
    query=query, max_results=20, sort_by=arxiv.SortCriterion.Relevance
)
```


```python
result_list = []

for result in search.results():
    result_dict = {}

    result_dict.update({"title": result.title})
    result_dict.update({"summary": result.summary})

    # Taking the first url provided
    result_dict.update({"article_url": [x.href for x in result.links][0]})
    result_dict.update({"pdf_url": [x.href for x in result.links][1]})
    result_list.append(result_dict)
```


```python
result_list[0]
```




    {'title': 'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features',
     'summary': 'Models based on large-pretrained language models, such as S(entence)BERT,\nprovide effective and efficient sentence embeddings that show high correlation\nto human similarity ratings, but lack interpretability. On the other hand,\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\nRepresentation, AMR) can make explicit the semantic aspects in which two\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\nand do not reach state-of-the-art performance when rating sentence similarity.\n  In this work, we aim at the best of both worlds, by learning to induce\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\nvarious semantic sentence features (e.g., semantic roles, negation, or\nquantification). We show how to i) learn a decomposition of the sentence\nembeddings into semantic features, through approximation of a suite of\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\nthe neural embeddings by controlling the decomposition learning process with a\nsecond objective that enforces consistency with the similarity ratings of an\nSBERT teacher model. In our experimental studies, we show that our approach\noffers interpretability -- while fully preserving the effectiveness and\nefficiency of the neural sentence embeddings.',
     'article_url': 'http://arxiv.org/abs/2206.07023v2',
     'pdf_url': 'http://arxiv.org/pdf/2206.07023v2'}


key: _node_content
string_value:
{"id_": "d3def4ee-a9fa-495e-960d-4900d0114fd0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9daa6650-83d9-469d-a903-178e3bd488ae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ad925f1a88c40cf8356899a0f5b35f21d50f605d7f98d80ca99db328e9674db9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5b5f7bc-632c-41f5-b469-a07f82fd1a58", "node_type": "1", "metadata": {}, "hash": "c69f519d93f62a0553c1392de1d6e130e8779161aea688eb237db3e36ead483f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3685, "end_char_idx": 7209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
result_list[0]
```




    {'title': 'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features',
     'summary': 'Models based on large-pretrained language models, such as S(entence)BERT,\nprovide effective and efficient sentence embeddings that show high correlation\nto human similarity ratings, but lack interpretability. On the other hand,\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\nRepresentation, AMR) can make explicit the semantic aspects in which two\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\nand do not reach state-of-the-art performance when rating sentence similarity.\n  In this work, we aim at the best of both worlds, by learning to induce\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\nvarious semantic sentence features (e.g., semantic roles, negation, or\nquantification). We show how to i) learn a decomposition of the sentence\nembeddings into semantic features, through approximation of a suite of\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\nthe neural embeddings by controlling the decomposition learning process with a\nsecond objective that enforces consistency with the similarity ratings of an\nSBERT teacher model. In our experimental studies, we show that our approach\noffers interpretability -- while fully preserving the effectiveness and\nefficiency of the neural sentence embeddings.',
     'article_url': 'http://arxiv.org/abs/2206.07023v2',
     'pdf_url': 'http://arxiv.org/pdf/2206.07023v2'}




```python
for i, result in enumerate(result_list):
    print(f"{i + 1}: {result['title']}")
```

    1: SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features
    2: Are Classes Clusters?
    3: Semantic Composition in Visually Grounded Language Models
    4: Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions
    5: Learning Probabilistic Sentence Representations from Paraphrases
    6: Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings
    7: How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation
    8: Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences
    9: Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
    10: Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings
    11: SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding
    12: Learning Joint Representations of Videos and Sentences with Web Image Search
    13: Character-based Neural Networks for Sentence Pair Modeling
    14: Train Once, Test Anywhere: Zero-Shot Learning for Text Classification
    15: Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models
    16: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
    17: In Search for Linear Relations in Sentence Embedding Spaces
    18: Learning to Borrow -- Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion
    19: Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences
    20: Relational Sentence Embedding for Flexible Semantic Matching


key: _node_content
string_value:
{"id_": "c5b5f7bc-632c-41f5-b469-a07f82fd1a58", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3def4ee-a9fa-495e-960d-4900d0114fd0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6c85f2e494023820356bce1e0d6cc1a2f89019d4b28399c4e1d8691ab132b065", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "728e18ab-01e0-4c55-bc48-9bf03837507f", "node_type": "1", "metadata": {}, "hash": "5551b0d12a201998e13de442ff7e88af66237ff9e6e8c42df97a98906f654f26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7212, "end_char_idx": 10568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Cross-encoder

We'll create a cross-encoder using the ```Completions``` endpoint - the key factors to consider here are:
- Make your examples domain-specific - the strength of cross-encoders comes when you tailor them to your domain.
- There is a trade-off between how many potential examples to re-rank vs. processing speed. Consider batching and parallel processing cross-encoder requests to process them more quickly.

The steps here are:
- Build a prompt to assess relevance and provide few-shot examples to tune it to your domain.
- Add a ```logit bias``` for the tokens for ``` Yes``` and ``` No``` to decrease the likelihood of any other tokens occurring.
- Return the classification of yes/no as well as the ```logprobs```.
- Rerank the results by the ```logprobs``` keyed on ``` Yes```.


```python
tokens = [" Yes", " No"]
tokenizer = tiktoken.encoding_for_model(OPENAI_MODEL)
ids = [tokenizer.encode(token) for token in tokens]
ids[0], ids[1]
```




    ([3363], [1400])




```python
prompt = '''
You are an Assistant responsible for helping detect whether the retrieved document is relevant to the query. For a given input, you need to output a single token: "Yes" or "No" indicating the retrieved document is relevant to the query.

Query: How to plant a tree?
Document: """Cars were invented in 1886, when German inventor Carl Benz patented his Benz Patent-Motorwagen.[3][4][5] Cars became widely available during the 20th century. One of the first cars affordable by the masses was the 1908 Model T, an American car manufactured by the Ford Motor Company. Cars were rapidly adopted in the US, where they replaced horse-drawn carriages.[6] In Europe and other parts of the world, demand for automobiles did not increase until after World War II.[7] The car is considered an essential part of the developed economy."""
Relevant: No

Query: Has the coronavirus vaccine been approved?
Document: """The Pfizer-BioNTech COVID-19 vaccine was approved for emergency use in the United States on December 11, 2020."""
Relevant: Yes

Query: What is the capital of France?
Document: """Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Its 19th-century cityscape is crisscrossed by wide boulevards and the River Seine. Beyond such landmarks as the Eiffel Tower and the 12th-century, Gothic Notre-Dame cathedral, the city is known for its cafe culture and designer boutiques along the Rue du Faubourg Saint-Honoré."""
Relevant: Yes

Query: What are some papers to learn about PPO reinforcement learning?
Document: """Proximal Policy Optimization and its Dynamic Version for Sequence Generation: In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance."""


key: _node_content
string_value:
{"id_": "728e18ab-01e0-4c55-bc48-9bf03837507f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5b5f7bc-632c-41f5-b469-a07f82fd1a58", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9e270f76ce0ec33e201d5390232df333e62018df942f0a7333f2fc52e4483516", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7163bd0-b67c-46aa-abe1-6465da808d5d", "node_type": "1", "metadata": {}, "hash": "409ecad44c74d085069fde984bf6542b2ffdd783ed5867f1d9ab377171fa8448", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8788, "end_char_idx": 11847, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Cars were rapidly adopted in the US, where they replaced horse-drawn carriages.[6] In Europe and other parts of the world, demand for automobiles did not increase until after World War II.[7] The car is considered an essential part of the developed economy."""
Relevant: No

Query: Has the coronavirus vaccine been approved?
Document: """The Pfizer-BioNTech COVID-19 vaccine was approved for emergency use in the United States on December 11, 2020."""
Relevant: Yes

Query: What is the capital of France?
Document: """Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Its 19th-century cityscape is crisscrossed by wide boulevards and the River Seine. Beyond such landmarks as the Eiffel Tower and the 12th-century, Gothic Notre-Dame cathedral, the city is known for its cafe culture and designer boutiques along the Rue du Faubourg Saint-Honoré."""
Relevant: Yes

Query: What are some papers to learn about PPO reinforcement learning?
Document: """Proximal Policy Optimization and its Dynamic Version for Sequence Generation: In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance."""
Relevant: Yes

Query: Explain sentence embeddings
Document: """Inside the bubble: exploring the environments of reionisation-era Lyman-α emitting galaxies with JADES and FRESCO: We present a study of the environments of 16 Lyman-α emitting galaxies (LAEs) in the reionisation era (5.8<z<8) identified by JWST/NIRSpec as part of the JWST Advanced Deep Extragalactic Survey (JADES). Unless situated in sufficiently (re)ionised regions, Lyman-α emission from these galaxies would be strongly absorbed by neutral gas in the intergalactic medium (IGM). We conservatively estimate sizes of the ionised regions required to reconcile the relatively low Lyman-α velocity offsets (ΔvLyα<300kms−1) with moderately high Lyman-α escape fractions (fesc,Lyα>5%) observed in our sample of LAEs, indicating the presence of ionised ``bubbles'' with physical sizes of the order of 0.1pMpc≲Rion≲1pMpc in a patchy reionisation scenario where the bubbles are embedded in a fully neutral IGM. Around half of the LAEs in our sample are found to coincide with large-scale galaxy overdensities seen in FRESCO at z∼5.8-5.9 and z∼7.3, suggesting Lyman-α transmission is strongly enhanced in such overdense regions, and underlining the importance of LAEs as tracers of the first large-scale ionised bubbles.


key: _node_content
string_value:
{"id_": "b7163bd0-b67c-46aa-abe1-6465da808d5d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "728e18ab-01e0-4c55-bc48-9bf03837507f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e539bddf44243b0954d8169c567d145a7619af44582a6e6a4f6ae40b004c28fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e", "node_type": "1", "metadata": {}, "hash": "c5fbe626d8cacb2ca35eb6d8377665387dbf916d1d1093304e5c614c7bc72162", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10323, "end_char_idx": 13361, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance."""
Relevant: Yes

Query: Explain sentence embeddings
Document: """Inside the bubble: exploring the environments of reionisation-era Lyman-α emitting galaxies with JADES and FRESCO: We present a study of the environments of 16 Lyman-α emitting galaxies (LAEs) in the reionisation era (5.8<z<8) identified by JWST/NIRSpec as part of the JWST Advanced Deep Extragalactic Survey (JADES). Unless situated in sufficiently (re)ionised regions, Lyman-α emission from these galaxies would be strongly absorbed by neutral gas in the intergalactic medium (IGM). We conservatively estimate sizes of the ionised regions required to reconcile the relatively low Lyman-α velocity offsets (ΔvLyα<300kms−1) with moderately high Lyman-α escape fractions (fesc,Lyα>5%) observed in our sample of LAEs, indicating the presence of ionised ``bubbles'' with physical sizes of the order of 0.1pMpc≲Rion≲1pMpc in a patchy reionisation scenario where the bubbles are embedded in a fully neutral IGM. Around half of the LAEs in our sample are found to coincide with large-scale galaxy overdensities seen in FRESCO at z∼5.8-5.9 and z∼7.3, suggesting Lyman-α transmission is strongly enhanced in such overdense regions, and underlining the importance of LAEs as tracers of the first large-scale ionised bubbles. Considering only spectroscopically confirmed galaxies, we find our sample of UV-faint LAEs (MUV≳−20mag) and their direct neighbours are generally not able to produce the required ionised regions based on the Lyman-α transmission properties, suggesting lower-luminosity sources likely play an important role in carving out these bubbles. These observations demonstrate the combined power of JWST multi-object and slitless spectroscopy in acquiring a unique view of the early stages of Cosmic Reionisation via the most distant LAEs."""
Relevant: No

Query: {query}
Document: """{document}"""
Relevant:
'''


@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def document_relevance(query, document):
    response = openai.chat.completions.create(
        model="text-davinci-003",
        message=prompt.format(query=query, document=document),
        temperature=0,
        logprobs=True,
        logit_bias={3363: 1, 1400: 1},
    )

    return (
        query,
        document,
        response.choices[0].message.content,
        response.choices[0].logprobs.token_logprobs[0],
    )
```


```python
content = result_list[0]["title"] + ": " + result_list[0]["summary"]

# Set logprobs to 1 so our response will include the most probable token the model identified
response = openai.chat.completions.create(
    model=OPENAI_MODEL,
    prompt=prompt.format(query=query, document=content),
    temperature=0,
    logprobs=1,
    logit_bias={3363: 1, 1400: 1},
    max_tokens=1,
)
```


key: _node_content
string_value:
{"id_": "aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7163bd0-b67c-46aa-abe1-6465da808d5d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f4cde4fe9db3aee0a7d075fa6231802cd792b9625ceb2bf7c5cfe12b001baa5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49dbaf8e-de1c-42a9-8a2b-c700df543296", "node_type": "1", "metadata": {}, "hash": "e54dd7e71cf42f3fd9da720f616d855d4058a7f45248ca559200c5d92389c56d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11848, "end_char_idx": 14767, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Considering only spectroscopically confirmed galaxies, we find our sample of UV-faint LAEs (MUV≳−20mag) and their direct neighbours are generally not able to produce the required ionised regions based on the Lyman-α transmission properties, suggesting lower-luminosity sources likely play an important role in carving out these bubbles. These observations demonstrate the combined power of JWST multi-object and slitless spectroscopy in acquiring a unique view of the early stages of Cosmic Reionisation via the most distant LAEs."""
Relevant: No

Query: {query}
Document: """{document}"""
Relevant:
'''


@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def document_relevance(query, document):
    response = openai.chat.completions.create(
        model="text-davinci-003",
        message=prompt.format(query=query, document=document),
        temperature=0,
        logprobs=True,
        logit_bias={3363: 1, 1400: 1},
    )

    return (
        query,
        document,
        response.choices[0].message.content,
        response.choices[0].logprobs.token_logprobs[0],
    )
```


```python
content = result_list[0]["title"] + ": " + result_list[0]["summary"]

# Set logprobs to 1 so our response will include the most probable token the model identified
response = openai.chat.completions.create(
    model=OPENAI_MODEL,
    prompt=prompt.format(query=query, document=content),
    temperature=0,
    logprobs=1,
    logit_bias={3363: 1, 1400: 1},
    max_tokens=1,
)
```


```python
result = response.choices[0]
print(f"Result was {result.message.content}")
print(f"Logprobs was {result.logprobs.token_logprobs[0]}")
print("\nBelow is the full logprobs object\n\n")
print(result["logprobs"])
```

    Result was Yes
    Logprobs was -0.05869877
    
    Below is the full logprobs object
    
    
    {
      "tokens": [
        "Yes"
      ],
      "token_logprobs": [
        -0.05869877
      ],
      "top_logprobs": [
        {
          "Yes": -0.05869877
        }
      ],
      "text_offset": [
        5764
      ]
    }



```python
output_list = []
for x in result_list:
    content = x["title"] + ": " + x["summary"]

    try:
        output_list.append(document_relevance(query, document=content))

    except Exception as e:
        print(e)
```


```python
output_list[:10]
```




    [('how do bi-encoders work for sentence embeddings',
      'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features: Models based on large-pretrained language models, such as S(entence)BERT,\nprovide effective and efficient sentence embeddings that show high correlation\nto human similarity ratings, but lack interpretability. On the other hand,\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\nRepresentation, AMR) can make explicit the semantic aspects in which two\nsentences are similar.


key: _node_content
string_value:
{"id_": "49dbaf8e-de1c-42a9-8a2b-c700df543296", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "628ca83a6f66725458c095551d56779191dcb081923d6cc2099251f30c8fede1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95265b2e-db56-427f-a23c-ee3f3f64db13", "node_type": "1", "metadata": {}, "hash": "69b57bbe8b80f121f619b94de8a976849fd9ddc89d99f2d968ac880bb716bd0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13364, "end_char_idx": 16421, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
result = response.choices[0]
print(f"Result was {result.message.content}")
print(f"Logprobs was {result.logprobs.token_logprobs[0]}")
print("\nBelow is the full logprobs object\n\n")
print(result["logprobs"])
```

    Result was Yes
    Logprobs was -0.05869877
    
    Below is the full logprobs object
    
    
    {
      "tokens": [
        "Yes"
      ],
      "token_logprobs": [
        -0.05869877
      ],
      "top_logprobs": [
        {
          "Yes": -0.05869877
        }
      ],
      "text_offset": [
        5764
      ]
    }



```python
output_list = []
for x in result_list:
    content = x["title"] + ": " + x["summary"]

    try:
        output_list.append(document_relevance(query, document=content))

    except Exception as e:
        print(e)
```


```python
output_list[:10]
```




    [('how do bi-encoders work for sentence embeddings',
      'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features: Models based on large-pretrained language models, such as S(entence)BERT,\nprovide effective and efficient sentence embeddings that show high correlation\nto human similarity ratings, but lack interpretability. On the other hand,\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\nRepresentation, AMR) can make explicit the semantic aspects in which two\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\nand do not reach state-of-the-art performance when rating sentence similarity.\n  In this work, we aim at the best of both worlds, by learning to induce\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\nvarious semantic sentence features (e.g., semantic roles, negation, or\nquantification). We show how to i) learn a decomposition of the sentence\nembeddings into semantic features, through approximation of a suite of\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\nthe neural embeddings by controlling the decomposition learning process with a\nsecond objective that enforces consistency with the similarity ratings of an\nSBERT teacher model. In our experimental studies, we show that our approach\noffers interpretability -- while fully preserving the effectiveness and\nefficiency of the neural sentence embeddings.',
      'Yes',
      -0.05326408),
     ('how do bi-encoders work for sentence embeddings',
      'Are Classes Clusters?: Sentence embedding models aim to provide general purpose embeddings for\nsentences. Most of the models studied in this paper claim to perform well on\nSTS tasks - but they do not report on their suitability for clustering. This\npaper looks at four recent sentence embedding models (Universal Sentence\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\nbrief overview of the ideas behind their implementations.


key: _node_content
string_value:
{"id_": "95265b2e-db56-427f-a23c-ee3f3f64db13", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49dbaf8e-de1c-42a9-8a2b-c700df543296", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "711e596a43851da6e09ef3bcb6740ad2f60ab33c3fdc4749922228fe84b4ac75", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a288d6e-1e5e-4a5f-a801-d89693cd6dc5", "node_type": "1", "metadata": {}, "hash": "ec57d9ad7b08d80c894e8827b95c8b79aea64cd57921ff8a288b8515813212bd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14768, "end_char_idx": 17738, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, such metrics tend to be slow, rely on parsers,\nand do not reach state-of-the-art performance when rating sentence similarity.\n  In this work, we aim at the best of both worlds, by learning to induce\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\nvarious semantic sentence features (e.g., semantic roles, negation, or\nquantification). We show how to i) learn a decomposition of the sentence\nembeddings into semantic features, through approximation of a suite of\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\nthe neural embeddings by controlling the decomposition learning process with a\nsecond objective that enforces consistency with the similarity ratings of an\nSBERT teacher model. In our experimental studies, we show that our approach\noffers interpretability -- while fully preserving the effectiveness and\nefficiency of the neural sentence embeddings.',
      'Yes',
      -0.05326408),
     ('how do bi-encoders work for sentence embeddings',
      'Are Classes Clusters?: Sentence embedding models aim to provide general purpose embeddings for\nsentences. Most of the models studied in this paper claim to perform well on\nSTS tasks - but they do not report on their suitability for clustering. This\npaper looks at four recent sentence embedding models (Universal Sentence\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\nbrief overview of the ideas behind their implementations. It then investigates\nhow well topic classes in two text classification datasets (Amazon Reviews (Ni\net al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their\ncorresponding sentence embedding space. While the performance of the resulting\nclassification model is far from perfect, it is better than random. This is\ninteresting because the classification model has been constructed in an\nunsupervised way. The topic classes in these real life topic classification\ndatasets can be partly reconstructed by clustering the corresponding sentence\nembeddings.',
      'No',
      -0.009535169),
     ('how do bi-encoders work for sentence embeddings',
      "Semantic Composition in Visually Grounded Language Models: What is sentence meaning and its ideal representation? Much of the expressive\npower of human language derives from semantic composition, the mind's ability\nto represent meaning hierarchically & relationally over constituents. At the\nsame time, much sentential meaning is outside the text and requires grounding\nin sensory, motor, and experiential modalities to be adequately learned.\nAlthough large language models display considerable compositional ability,\nrecent work shows that visually-grounded language models drastically fail to\nrepresent compositional structure.


key: _node_content
string_value:
{"id_": "4a288d6e-1e5e-4a5f-a801-d89693cd6dc5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95265b2e-db56-427f-a23c-ee3f3f64db13", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d43950eada597d280fe2fa812ae77c64dfcab6f37345ed66e1d3d2b2a9fc2c4e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6", "node_type": "1", "metadata": {}, "hash": "bf5010e8448d5116cea8ea74ee9ce14367937d7d88bc31f3ce02ad2060ecd3cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15903, "end_char_idx": 19480, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
: Sentence embedding models aim to provide general purpose embeddings for\nsentences. Most of the models studied in this paper claim to perform well on\nSTS tasks - but they do not report on their suitability for clustering. This\npaper looks at four recent sentence embedding models (Universal Sentence\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\nbrief overview of the ideas behind their implementations. It then investigates\nhow well topic classes in two text classification datasets (Amazon Reviews (Ni\net al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their\ncorresponding sentence embedding space. While the performance of the resulting\nclassification model is far from perfect, it is better than random. This is\ninteresting because the classification model has been constructed in an\nunsupervised way. The topic classes in these real life topic classification\ndatasets can be partly reconstructed by clustering the corresponding sentence\nembeddings.',
      'No',
      -0.009535169),
     ('how do bi-encoders work for sentence embeddings',
      "Semantic Composition in Visually Grounded Language Models: What is sentence meaning and its ideal representation? Much of the expressive\npower of human language derives from semantic composition, the mind's ability\nto represent meaning hierarchically & relationally over constituents. At the\nsame time, much sentential meaning is outside the text and requires grounding\nin sensory, motor, and experiential modalities to be adequately learned.\nAlthough large language models display considerable compositional ability,\nrecent work shows that visually-grounded language models drastically fail to\nrepresent compositional structure. In this thesis, we explore whether & how\nmodels compose visually grounded semantics, and how we might improve their\nability to do so.\n  Specifically, we introduce 1) WinogroundVQA, a new compositional visual\nquestion answering benchmark, 2) Syntactic Neural Module Distillation, a\nmeasure of compositional ability in sentence embedding models, 3) Causal\nTracing for Image Captioning Models to locate neural representations vital for\nvision-language composition, 4) Syntactic MeanPool to inject a compositional\ninductive bias into sentence embeddings, and 5) Cross-modal Attention\nCongruence Regularization, a self-supervised objective function for\nvision-language relation alignment. We close by discussing connections of our\nwork to neuroscience, psycholinguistics, formal semantics, and philosophy.",
      'No',
      -0.008887106),
     ('how do bi-encoders work for sentence embeddings',
      "Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions: Text embedding models from Natural Language Processing can map text data\n(e.g. words, sentences, documents) to supposedly meaningful numerical\nrepresentations (a.k.a. text embeddings). While such models are increasingly\napplied in social science research, one important issue is often not addressed:\nthe extent to which these embeddings are valid representations of constructs\nrelevant for social science research. We therefore propose the use of the\nclassic construct validity framework to evaluate the validity of text\nembeddings. We show how this framework can be adapted to the opaque and\nhigh-dimensional nature of text embeddings, with application to survey\nquestions. We include several popular text embedding methods (e.g.


key: _node_content
string_value:
{"id_": "ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a288d6e-1e5e-4a5f-a801-d89693cd6dc5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "20fcad9ddd2da10f51c10cc7c27950361df0fa4874bc420e14752286e0cb487f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f", "node_type": "1", "metadata": {}, "hash": "b2f1b8ba08f4d4366ce16ecff5edb83fd9c2338dbd04f63975f3f6485c6bf97d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17739, "end_char_idx": 21470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this thesis, we explore whether & how\nmodels compose visually grounded semantics, and how we might improve their\nability to do so.\n  Specifically, we introduce 1) WinogroundVQA, a new compositional visual\nquestion answering benchmark, 2) Syntactic Neural Module Distillation, a\nmeasure of compositional ability in sentence embedding models, 3) Causal\nTracing for Image Captioning Models to locate neural representations vital for\nvision-language composition, 4) Syntactic MeanPool to inject a compositional\ninductive bias into sentence embeddings, and 5) Cross-modal Attention\nCongruence Regularization, a self-supervised objective function for\nvision-language relation alignment. We close by discussing connections of our\nwork to neuroscience, psycholinguistics, formal semantics, and philosophy.",
      'No',
      -0.008887106),
     ('how do bi-encoders work for sentence embeddings',
      "Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions: Text embedding models from Natural Language Processing can map text data\n(e.g. words, sentences, documents) to supposedly meaningful numerical\nrepresentations (a.k.a. text embeddings). While such models are increasingly\napplied in social science research, one important issue is often not addressed:\nthe extent to which these embeddings are valid representations of constructs\nrelevant for social science research. We therefore propose the use of the\nclassic construct validity framework to evaluate the validity of text\nembeddings. We show how this framework can be adapted to the opaque and\nhigh-dimensional nature of text embeddings, with application to survey\nquestions. We include several popular text embedding methods (e.g. fastText,\nGloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct\nvalidity analyses. We find evidence of convergent and discriminant validity in\nsome cases. We also show that embeddings can be used to predict respondent's\nanswers to completely new survey questions. Furthermore, BERT-based embedding\ntechniques and the Universal Sentence Encoder provide more valid\nrepresentations of survey questions than do others. Our results thus highlight\nthe necessity to examine the construct validity of text embeddings before\ndeploying them in social science research.",
      'No',
      -0.008583762),
     ('how do bi-encoders work for sentence embeddings',
      'Learning Probabilistic Sentence Representations from Paraphrases: Probabilistic word embeddings have shown effectiveness in capturing notions\nof generality and entailment, but there is very little work on doing the\nanalogous type of investigation for sentences. In this paper we define\nprobabilistic models that produce distributions for sentences. Our\nbest-performing model treats each word as a linear transformation operator\napplied to a multivariate Gaussian distribution. We train our models on\nparaphrases and demonstrate that they naturally capture sentence specificity.\nWhile our proposed model achieves the best performance overall, we also show\nthat specificity is represented by simpler architectures via the norm of the\nsentence vectors. Qualitative analysis shows that our probabilistic model\ncaptures sentential entailment and provides ways to analyze the specificity and\npreciseness of individual words.',
      'No',
      -0.011975748),
     ('how do bi-encoders work for sentence embeddings',
      "Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings: Semantic sentence embeddings are usually supervisedly built minimizing\ndistances between pairs of embeddings of sentences labelled as semantically\nsimilar by annotators.


key: _node_content
string_value:
{"id_": "2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a17e4cabf04e6ce3f62d7743dcf5db38b8c1ad0adff7a0faa38a33a861d00a1e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1f42216-25fc-41b9-b3b5-b0783dcc8df0", "node_type": "1", "metadata": {}, "hash": "78dbfb73c3c8ccf7ea9995913b7e9cd63c473a09dfbf60eb0deacbae86cf9cfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19585, "end_char_idx": 23318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We find evidence of convergent and discriminant validity in\nsome cases. We also show that embeddings can be used to predict respondent's\nanswers to completely new survey questions. Furthermore, BERT-based embedding\ntechniques and the Universal Sentence Encoder provide more valid\nrepresentations of survey questions than do others. Our results thus highlight\nthe necessity to examine the construct validity of text embeddings before\ndeploying them in social science research.",
      'No',
      -0.008583762),
     ('how do bi-encoders work for sentence embeddings',
      'Learning Probabilistic Sentence Representations from Paraphrases: Probabilistic word embeddings have shown effectiveness in capturing notions\nof generality and entailment, but there is very little work on doing the\nanalogous type of investigation for sentences. In this paper we define\nprobabilistic models that produce distributions for sentences. Our\nbest-performing model treats each word as a linear transformation operator\napplied to a multivariate Gaussian distribution. We train our models on\nparaphrases and demonstrate that they naturally capture sentence specificity.\nWhile our proposed model achieves the best performance overall, we also show\nthat specificity is represented by simpler architectures via the norm of the\nsentence vectors. Qualitative analysis shows that our probabilistic model\ncaptures sentential entailment and provides ways to analyze the specificity and\npreciseness of individual words.',
      'No',
      -0.011975748),
     ('how do bi-encoders work for sentence embeddings',
      "Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings: Semantic sentence embeddings are usually supervisedly built minimizing\ndistances between pairs of embeddings of sentences labelled as semantically\nsimilar by annotators. Since big labelled datasets are rare, in particular for\nnon-English languages, and expensive, recent studies focus on unsupervised\napproaches that require not-paired input sentences. We instead propose a\nlanguage-independent approach to build large datasets of pairs of informal\ntexts weakly similar, without manual human effort, exploiting Twitter's\nintrinsic powerful signals of relatedness: replies and quotes of tweets. We use\nthe collected pairs to train a Transformer model with triplet-like structures,\nand we test the generated embeddings on Twitter NLP similarity tasks (PIT and\nTURL) and STSb. We also introduce four new sentence ranking evaluation\nbenchmarks of informal texts, carefully extracted from the initial collections\nof tweets, proving not only that our best model learns classical Semantic\nTextual Similarity, but also excels on tasks where pairs of sentences are not\nexact paraphrases. Ablation studies reveal how increasing the corpus size\ninfluences positively the results, even at 2M samples, suggesting that bigger\ncollections of Tweets still do not contain redundant information about semantic\nsimilarities.",
      'No',
      -0.01219046),
     ('how do bi-encoders work for sentence embeddings',
      "How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation: Sentence encoders map sentences to real valued vectors for use in downstream\napplications. To peek into these representations - e.g., to increase\ninterpretability of their results - probing tasks have been designed which\nquery them for linguistic knowledge. However, designing probing tasks for\nlesser-resourced languages is tricky, because these often lack large-scale\nannotated data or (high-quality) dependency parsers as a prerequisite of\nprobing task design in English.


key: _node_content
string_value:
{"id_": "a1f42216-25fc-41b9-b3b5-b0783dcc8df0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8cdcec10b099acfc43ab94d954bb17897bd20d149e570c5644f5cd7dbb5c3a85", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e3f285e-a756-406a-8ace-f26d91dfc2d2", "node_type": "1", "metadata": {}, "hash": "6b03c60e3055708922808458659a58a182a85c9d075f314ee08e216ceb7112b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21471, "end_char_idx": 25106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since big labelled datasets are rare, in particular for\nnon-English languages, and expensive, recent studies focus on unsupervised\napproaches that require not-paired input sentences. We instead propose a\nlanguage-independent approach to build large datasets of pairs of informal\ntexts weakly similar, without manual human effort, exploiting Twitter's\nintrinsic powerful signals of relatedness: replies and quotes of tweets. We use\nthe collected pairs to train a Transformer model with triplet-like structures,\nand we test the generated embeddings on Twitter NLP similarity tasks (PIT and\nTURL) and STSb. We also introduce four new sentence ranking evaluation\nbenchmarks of informal texts, carefully extracted from the initial collections\nof tweets, proving not only that our best model learns classical Semantic\nTextual Similarity, but also excels on tasks where pairs of sentences are not\nexact paraphrases. Ablation studies reveal how increasing the corpus size\ninfluences positively the results, even at 2M samples, suggesting that bigger\ncollections of Tweets still do not contain redundant information about semantic\nsimilarities.",
      'No',
      -0.01219046),
     ('how do bi-encoders work for sentence embeddings',
      "How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation: Sentence encoders map sentences to real valued vectors for use in downstream\napplications. To peek into these representations - e.g., to increase\ninterpretability of their results - probing tasks have been designed which\nquery them for linguistic knowledge. However, designing probing tasks for\nlesser-resourced languages is tricky, because these often lack large-scale\nannotated data or (high-quality) dependency parsers as a prerequisite of\nprobing task design in English. To investigate how to probe sentence embeddings\nin such cases, we investigate sensitivity of probing task results to structural\ndesign choices, conducting the first such large scale study. We show that\ndesign choices like size of the annotated probing dataset and type of\nclassifier used for evaluation do (sometimes substantially) influence probing\noutcomes. We then probe embeddings in a multilingual setup with design choices\nthat lie in a 'stable region', as we identify for English, and find that\nresults on English do not transfer to other languages. Fairer and more\ncomprehensive sentence-level probing evaluation should thus be carried out on\nmultiple languages in the future.",
      'No',
      -0.015550519),
     ('how do bi-encoders work for sentence embeddings',
      'Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences: Sentence embedding methods offer a powerful approach for working with short\ntextual constructs or sequences of words. By representing sentences as dense\nnumerical vectors, many natural language processing (NLP) applications have\nimproved their performance. However, relatively little is understood about the\nlatent structure of sentence embeddings. Specifically, research has not\naddressed whether the length and structure of sentences impact the sentence\nembedding space and topology. This paper reports research on a set of\ncomprehensive clustering and network analyses targeting sentence and\nsub-sentence embedding spaces. Results show that one method generates the most\nclusterable embeddings. In general, the embeddings of span sub-sentences have\nbetter clustering properties than the original sentences. The results have\nimplications for future sentence embedding models and applications.


key: _node_content
string_value:
{"id_": "7e3f285e-a756-406a-8ace-f26d91dfc2d2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1f42216-25fc-41b9-b3b5-b0783dcc8df0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "518cf1bc2985e640bab5da25b367137270d29d069c4cf08a1997359de35db27e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7c7675b-3409-4f87-b7fe-d9208dfe797d", "node_type": "1", "metadata": {}, "hash": "c46a8681ab2e68bb10f4c7f45f736d3412347c95bf944cd65ecc37b6dabca402", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23099, "end_char_idx": 26787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, designing probing tasks for\nlesser-resourced languages is tricky, because these often lack large-scale\nannotated data or (high-quality) dependency parsers as a prerequisite of\nprobing task design in English. To investigate how to probe sentence embeddings\nin such cases, we investigate sensitivity of probing task results to structural\ndesign choices, conducting the first such large scale study. We show that\ndesign choices like size of the annotated probing dataset and type of\nclassifier used for evaluation do (sometimes substantially) influence probing\noutcomes. We then probe embeddings in a multilingual setup with design choices\nthat lie in a 'stable region', as we identify for English, and find that\nresults on English do not transfer to other languages. Fairer and more\ncomprehensive sentence-level probing evaluation should thus be carried out on\nmultiple languages in the future.",
      'No',
      -0.015550519),
     ('how do bi-encoders work for sentence embeddings',
      'Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences: Sentence embedding methods offer a powerful approach for working with short\ntextual constructs or sequences of words. By representing sentences as dense\nnumerical vectors, many natural language processing (NLP) applications have\nimproved their performance. However, relatively little is understood about the\nlatent structure of sentence embeddings. Specifically, research has not\naddressed whether the length and structure of sentences impact the sentence\nembedding space and topology. This paper reports research on a set of\ncomprehensive clustering and network analyses targeting sentence and\nsub-sentence embedding spaces. Results show that one method generates the most\nclusterable embeddings. In general, the embeddings of span sub-sentences have\nbetter clustering properties than the original sentences. The results have\nimplications for future sentence embedding models and applications.',
      'No',
      -0.012663184),
     ('how do bi-encoders work for sentence embeddings',
      'Vec2Sent: Probing Sentence Embeddings with Natural Language Generation: We introspect black-box sentence embeddings by conditionally generating from\nthem with the objective to retrieve the underlying discrete sentence. We\nperceive of this as a new unsupervised probing task and show that it correlates\nwell with downstream task performance. We also illustrate how the language\ngenerated from different encoders differs. We apply our approach to generate\nsentence analogies from sentence embeddings.',
      'Yes',
      -0.004863006),
     ('how do bi-encoders work for sentence embeddings',
      'Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings: Semantic representation learning for sentences is an important and\nwell-studied problem in NLP. The current trend for this task involves training\na Transformer-based sentence encoder through a contrastive objective with text,\ni.e., clustering sentences with semantically similar meanings and scattering\nothers. In this work, we find the performance of Transformer models as sentence\nencoders can be improved by training with multi-modal multi-task losses, using\nunpaired examples from another modality (e.g., sentences and unrelated\nimage/audio data). In particular, besides learning by the contrastive loss on\ntext, our model clusters examples from a non-linguistic domain (e.g.,\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\nour framework on unpaired non-linguistic data makes it language-agnostic,\nenabling it to be widely applicable beyond English NLP.


key: _node_content
string_value:
{"id_": "d7c7675b-3409-4f87-b7fe-d9208dfe797d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e3f285e-a756-406a-8ace-f26d91dfc2d2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d9346aa1132b00a7cae27914f18b5ef8a14d3bb82064664518236431e63ece3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fd4a3bf-16a0-4d17-ac17-eceda84375c6", "node_type": "1", "metadata": {}, "hash": "b77b059a6e4a6df08101daec25b902fbeabe859cb7633f927e152a7795d191ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24908, "end_char_idx": 27675, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In general, the embeddings of span sub-sentences have\nbetter clustering properties than the original sentences. The results have\nimplications for future sentence embedding models and applications.',
      'No',
      -0.012663184),
     ('how do bi-encoders work for sentence embeddings',
      'Vec2Sent: Probing Sentence Embeddings with Natural Language Generation: We introspect black-box sentence embeddings by conditionally generating from\nthem with the objective to retrieve the underlying discrete sentence. We\nperceive of this as a new unsupervised probing task and show that it correlates\nwell with downstream task performance. We also illustrate how the language\ngenerated from different encoders differs. We apply our approach to generate\nsentence analogies from sentence embeddings.',
      'Yes',
      -0.004863006),
     ('how do bi-encoders work for sentence embeddings',
      'Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings: Semantic representation learning for sentences is an important and\nwell-studied problem in NLP. The current trend for this task involves training\na Transformer-based sentence encoder through a contrastive objective with text,\ni.e., clustering sentences with semantically similar meanings and scattering\nothers. In this work, we find the performance of Transformer models as sentence\nencoders can be improved by training with multi-modal multi-task losses, using\nunpaired examples from another modality (e.g., sentences and unrelated\nimage/audio data). In particular, besides learning by the contrastive loss on\ntext, our model clusters examples from a non-linguistic domain (e.g.,\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\nour framework on unpaired non-linguistic data makes it language-agnostic,\nenabling it to be widely applicable beyond English NLP. Experiments on 7\nsemantic textual similarity benchmarks reveal that models trained with the\nadditional non-linguistic (/images/audio) contrastive objective lead to higher\nquality sentence embeddings. This indicates that Transformer models are able to\ngeneralize better by doing a similar task (i.e., clustering) with unpaired\nexamples from different modalities in a multi-task fashion.',
      'No',
      -0.013869206)]




```python
output_df = pd.DataFrame(
    output_list, columns=["query", "document", "prediction", "logprobs"]
).reset_index()
# Use exp() to convert logprobs into probability
output_df["probability"] = output_df["logprobs"].apply(exp)
# Reorder based on likelihood of being Yes
output_df["yes_probability"] = output_df.apply(
    lambda x: x["probability"] * -1 + 1
    if x["prediction"] == "No"
    else x["probability"],
    axis=1,
)
output_df.head()
```


key: _node_content
string_value:
{"id_": "3fd4a3bf-16a0-4d17-ac17-eceda84375c6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7c7675b-3409-4f87-b7fe-d9208dfe797d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e11556d4096f38584f323a10b91a9f5d29b45e4cc3f32ff3b4472bd370efe6e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec", "node_type": "1", "metadata": {}, "hash": "55647bea4fbc3f70bf955f21af430a006d62dfe440b5afa05380b7569874f93d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25983, "end_char_idx": 29606, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The current trend for this task involves training\na Transformer-based sentence encoder through a contrastive objective with text,\ni.e., clustering sentences with semantically similar meanings and scattering\nothers. In this work, we find the performance of Transformer models as sentence\nencoders can be improved by training with multi-modal multi-task losses, using\nunpaired examples from another modality (e.g., sentences and unrelated\nimage/audio data). In particular, besides learning by the contrastive loss on\ntext, our model clusters examples from a non-linguistic domain (e.g.,\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\nour framework on unpaired non-linguistic data makes it language-agnostic,\nenabling it to be widely applicable beyond English NLP. Experiments on 7\nsemantic textual similarity benchmarks reveal that models trained with the\nadditional non-linguistic (/images/audio) contrastive objective lead to higher\nquality sentence embeddings. This indicates that Transformer models are able to\ngeneralize better by doing a similar task (i.e., clustering) with unpaired\nexamples from different modalities in a multi-task fashion.',
      'No',
      -0.013869206)]




```python
output_df = pd.DataFrame(
    output_list, columns=["query", "document", "prediction", "logprobs"]
).reset_index()
# Use exp() to convert logprobs into probability
output_df["probability"] = output_df["logprobs"].apply(exp)
# Reorder based on likelihood of being Yes
output_df["yes_probability"] = output_df.apply(
    lambda x: x["probability"] * -1 + 1
    if x["prediction"] == "No"
    else x["probability"],
    axis=1,
)
output_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>index</th>
      <th>query</th>
      <th>document</th>
      <th>prediction</th>
      <th>logprobs</th>
      <th>probability</th>
      <th>yes_probability</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>SBERT studies Meaning Representations: Decompo...</td>
      <td>Yes</td>
      <td>-0.053264</td>
      <td>0.948130</td>
      <td>0.948130</td>
    </tr>
    <tr>
      <th>1</th>
      <td>1</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Are Classes Clusters?: Sentence embedding mode...</td>
      <td>No</td>
      <td>-0.009535</td>
      <td>0.990510</td>
      <td>0.009490</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Semantic Composition in Visually Grounded Lang...</td>
      <td>No</td>
      <td>-0.008887</td>
      <td>0.991152</td>
      <td>0.008848</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Evaluating the Construct Validity of Text Embe...</td>
      <td>No</td>
      <td>-0.008584</td>
      <td>0.991453</td>
      <td>0.008547</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Learning Probabilistic Sentence Representation...</td>
      <td>No</td>
      <td>-0.011976</td>
      <td>0.988096</td>
      <td>0.011904</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3fd4a3bf-16a0-4d17-ac17-eceda84375c6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9bfc8045ce3c3ded0f640cf6cde48ef3c9102e18ecfb65d64a8019361830d21b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec762f01-0783-4631-a185-97e1f1908d57", "node_type": "1", "metadata": {}, "hash": "4301d325c747ff48cd51bab7eab09876a069081b5833546d979c5590bdb8e8be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Return reranked results
reranked_df = output_df.sort_values(
    by=["yes_probability"], ascending=False
).reset_index()
reranked_df.head(10)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>level_0</th>
      <th>index</th>
      <th>query</th>
      <th>document</th>
      <th>prediction</th>
      <th>logprobs</th>
      <th>probability</th>
      <th>yes_probability</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>16</td>
      <td>16</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>In Search for Linear Relations in Sentence Emb.</td>
      <td>Yes</td>
      <td>-0.004824</td>
      <td>0.995187</td>
      <td>0.995187</td>
    </tr>
    <tr>
      <th>1</th>
      <td>8</td>
      <td>8</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Vec2Sent: Probing Sentence Embeddings with Nat.</td>
      <td>Yes</td>
      <td>-0.004863</td>
      <td>0.995149</td>
      <td>0.995149</td>
    </tr>
    <tr>
      <th>2</th>
      <td>19</td>
      <td>19</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Relational Sentence Embedding for Flexible Sem.</td>
      <td>Yes</td>
      <td>-0.038814</td>
      <td>0.961930</td>
      <td>0.961930</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0</td>
      <td>0</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>SBERT studies Meaning Representations: Decompo.</td>
      <td>Yes</td>
      <td>-0.053264</td>
      <td>0.948130</td>
      <td>0.948130</td>
    </tr>
    <tr>
      <th>4</th>
      <td>15</td>
      <td>15</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Sentence-T5: Scalable Sentence Encoders from P.</td>
      <td>No</td>
      <td>-0.291893</td>
      <td>0.746849</td>
      <td>0.


key: _node_content
string_value:
{"id_": "ec762f01-0783-4631-a185-97e1f1908d57", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "34000819facf129eb5206551ad4e034ce37ef603ec6ede54252dfe6b08b1eec5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac21f0d2-c294-41b9-a77e-920fec71818d", "node_type": "1", "metadata": {}, "hash": "9ccd774dacbd5341f4ad9452c643df9af4b136c9c2a3c1e202c34a7c516725b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>Yes</td>
      <td>-0.004863</td>
      <td>0.995149</td>
      <td>0.995149</td>
    </tr>
    <tr>
      <th>2</th>
      <td>19</td>
      <td>19</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Relational Sentence Embedding for Flexible Sem.</td>
      <td>Yes</td>
      <td>-0.038814</td>
      <td>0.961930</td>
      <td>0.961930</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0</td>
      <td>0</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>SBERT studies Meaning Representations: Decompo.</td>
      <td>Yes</td>
      <td>-0.053264</td>
      <td>0.948130</td>
      <td>0.948130</td>
    </tr>
    <tr>
      <th>4</th>
      <td>15</td>
      <td>15</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Sentence-T5: Scalable Sentence Encoders from P.</td>
      <td>No</td>
      <td>-0.291893</td>
      <td>0.746849</td>
      <td>0.253151</td>
    </tr>
    <tr>
      <th>5</th>
      <td>6</td>
      <td>6</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>How to Probe Sentence Embeddings in Low-Resour.</td>
      <td>No</td>
      <td>-0.015551</td>
      <td>0.984570</td>
      <td>0.015430</td>
    </tr>
    <tr>
      <th>6</th>
      <td>18</td>
      <td>18</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Efficient and Flexible Topic Modeling using Pr.</td>
      <td>No</td>
      <td>-0.015296</td>
      <td>0.984820</td>
      <td>0.015180</td>
    </tr>
    <tr>
      <th>7</th>
      <td>9</td>
      <td>9</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Non-Linguistic Supervision for Contrastive Lea.</td>
      <td>No</td>
      <td>-0.013869</td>
      <td>0.986227</td>
      <td>0.013773</td>
    </tr>
    <tr>
      <th>8</th>
      <td>12</td>
      <td>12</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Character-based Neural Networks for Sentence P.</td>
      <td>No</td>
      <td>-0.


key: _node_content
string_value:
{"id_": "ac21f0d2-c294-41b9-a77e-920fec71818d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ec762f01-0783-4631-a185-97e1f1908d57", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cad0be18073284f57dea6e5086f0618627b25ca8d19e7e7deefb774874436e29", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cb5ac0c-7227-4fd7-8452-f32b2150f5b5", "node_type": "1", "metadata": {}, "hash": "c4a1cdfad550df49c77588a65fee24326cbbe17b705f2857cb71870de6044e47", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>No</td>
      <td>-0.015551</td>
      <td>0.984570</td>
      <td>0.015430</td>
    </tr>
    <tr>
      <th>6</th>
      <td>18</td>
      <td>18</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Efficient and Flexible Topic Modeling using Pr.</td>
      <td>No</td>
      <td>-0.015296</td>
      <td>0.984820</td>
      <td>0.015180</td>
    </tr>
    <tr>
      <th>7</th>
      <td>9</td>
      <td>9</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Non-Linguistic Supervision for Contrastive Lea.</td>
      <td>No</td>
      <td>-0.013869</td>
      <td>0.986227</td>
      <td>0.013773</td>
    </tr>
    <tr>
      <th>8</th>
      <td>12</td>
      <td>12</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Character-based Neural Networks for Sentence P.</td>
      <td>No</td>
      <td>-0.012866</td>
      <td>0.987216</td>
      <td>0.012784</td>
    </tr>
    <tr>
      <th>9</th>
      <td>7</td>
      <td>7</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Clustering and Network Analysis for the Embedd.</td>
      <td>No</td>
      <td>-0.012663</td>
      <td>0.987417</td>
      <td>0.012583</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Inspect our new top document following reranking
reranked_df["document"][0]
```




    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\nrepresentations of sentences. We acquire pairs of very similar sentences\ndiffering only by a small alterations (such as change of a noun, adding an\nadjective, noun or punctuation) from datasets for natural language inference\nusing a simple pattern method. We look into how such a small change within the\nsentence text affects its representation in the continuous space and how such\nalterations are reflected by some of the popular sentence embedding models. We\nfound that vector differences of some embeddings actually reflect small changes\nwithin a sentence.'


key: _node_content
string_value:
{"id_": "3cb5ac0c-7227-4fd7-8452-f32b2150f5b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac21f0d2-c294-41b9-a77e-920fec71818d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "193f23db2dded0ebc66e72d80775caa3b0f70366be282f0bfd6fae4f6bb165de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7d4cb0b-8214-497d-a25c-36e2d81f00ab", "node_type": "1", "metadata": {}, "hash": "b5e50adc8fdc354796302ff46de056e26161b30d28ff4177475e4807015d7615", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
986227</td>
      <td>0.013773</td>
    </tr>
    <tr>
      <th>8</th>
      <td>12</td>
      <td>12</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Character-based Neural Networks for Sentence P.</td>
      <td>No</td>
      <td>-0.012866</td>
      <td>0.987216</td>
      <td>0.012784</td>
    </tr>
    <tr>
      <th>9</th>
      <td>7</td>
      <td>7</td>
      <td>how do bi-encoders work for sentence embeddings</td>
      <td>Clustering and Network Analysis for the Embedd.</td>
      <td>No</td>
      <td>-0.012663</td>
      <td>0.987417</td>
      <td>0.012583</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Inspect our new top document following reranking
reranked_df["document"][0]
```




    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\nrepresentations of sentences. We acquire pairs of very similar sentences\ndiffering only by a small alterations (such as change of a noun, adding an\nadjective, noun or punctuation) from datasets for natural language inference\nusing a simple pattern method. We look into how such a small change within the\nsentence text affects its representation in the continuous space and how such\nalterations are reflected by some of the popular sentence embedding models. We\nfound that vector differences of some embeddings actually reflect small changes\nwithin a sentence.'



## Conclusion

We've shown how to create a tailored cross-encoder to rerank academic papers. This approach will work best where there are domain-specific nuances that can be used to pick the most relevant corpus for your users, and where some pre-filtering has taken place to limit the amount of data the cross-encoder will need to process. 

A few typical use cases we've seen are:
- Returning a list of 100 most relevant stock reports, then re-ordering into a top 5 or 10 based on the detailed context of a particular set of customer portfolios
- Running after a classic rules-based search that gets the top 100 or 1000 most relevant results to prune it according to a specific user's context


key: _node_content
string_value:
{"id_": "a7d4cb0b-8214-497d-a25c-36e2d81f00ab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3cb5ac0c-7227-4fd7-8452-f32b2150f5b5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e4ad151a18280c890dd5da569534bb8b198d9f85dbfc5d98023a38124875e86c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33132, "end_char_idx": 35818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>No</td>
      <td>-0.012663</td>
      <td>0.987417</td>
      <td>0.012583</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Inspect our new top document following reranking
reranked_df["document"][0]
```




    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\nrepresentations of sentences. We acquire pairs of very similar sentences\ndiffering only by a small alterations (such as change of a noun, adding an\nadjective, noun or punctuation) from datasets for natural language inference\nusing a simple pattern method. We look into how such a small change within the\nsentence text affects its representation in the continuous space and how such\nalterations are reflected by some of the popular sentence embedding models. We\nfound that vector differences of some embeddings actually reflect small changes\nwithin a sentence.'



## Conclusion

We've shown how to create a tailored cross-encoder to rerank academic papers. This approach will work best where there are domain-specific nuances that can be used to pick the most relevant corpus for your users, and where some pre-filtering has taken place to limit the amount of data the cross-encoder will need to process. 

A few typical use cases we've seen are:
- Returning a list of 100 most relevant stock reports, then re-ordering into a top 5 or 10 based on the detailed context of a particular set of customer portfolios
- Running after a classic rules-based search that gets the top 100 or 1000 most relevant results to prune it according to a specific user's context


### Taking this forward

Taking the few-shot approach, as we have here, can work well when the domain is general enough that a small number of examples will cover most reranking cases. However, as the differences between documents become more specific you may want to consider the ```Fine-tuning``` endpoint to make a more elaborate cross-encoder with a wider variety of examples.

There is also a latency impact of using ```text-davinci-003``` that you'll need to consider, with even our few examples above taking a couple seconds each - again, the ```Fine-tuning``` endpoint may help you here if you are able to get decent results from an ```ada``` or ```babbage``` fine-tuned model.

We've used the ```Completions``` endpoint from OpenAI to build our cross-encoder, but this area is well-served by the open-source community. [Here](https://huggingface.co/jeffwan/mmarco-mMiniLMv2-L12-H384-v1) is an example from HuggingFace, for example.

We hope you find this useful for tuning your search use cases, and look forward to seeing what you build.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb


key: _node_content
string_value:
{"id_": "da3e88ec-111f-4abd-b194-88ac4fb52157", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d1faed8-58e0-47b4-b542-ac24da4fe817", "node_type": "1", "metadata": {}, "hash": "b8a6f3ffc28b8f5102cba3eae6067a88e995678ef8fdaead23ed51c41fded080", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Developing Hallucination Guardrails

A guardrail is a set of rules and checks designed to ensure that the outputs of an LLM are accurate, appropriate, and aligned with user expectations. For more additional information on developing guardrails, you can refer to this [guide on developing guardrails](https://cookbook.openai.com/examples/how_to_use_guardrails).

In this notebook, we'll walk through the process of developing an output guardrail that specifically checks model outputs for hallucinations. 

This notebook will focus on:
1. Building out a strong eval set
2. Identifying specific criteria to measure hallucinations
3. Improving the accuracy of our guardrail with few-shot prompting



```python
from concurrent.futures import ThreadPoolExecutor
from IPython.display import display, HTML
import json
import pandas as pd
from sklearn.metrics import precision_score, recall_score
from typing import List
from openai import OpenAI

client = OpenAI()
```


```python
# Function to set up display options for pandas
def setup_pandas_display():
    # Increase display limits
    pd.set_option('display.max_rows', 500)
    pd.set_option('display.max_columns', 500)

# Function to make DataFrame scrollable in the notebook output
def make_scrollable(df):
    style = (
        '<style>'
        'div.output_scroll {'
        'resize: both;'
        'overflow: auto;'
        '}'
        '</style>'
    )
    html = f"{style}{df.to_html()}"
    display(HTML(html))

# Main function to display DataFrame
def display_dataframe(df):
    setup_pandas_display()    # Enable scrollable view
    make_scrollable(df)

```

## 1. Building out an eval set

Imagine we are a customer support team that is building out an automated support agent. We will be feeding the assistant information from our knowledge base about a specific set of policies for how to handle tickets such as returns, refunds, feedback, and expect the model to follow the policy when interacting with customers.

The first thing we will do is use GPT-4o to build out a set of policies that we will want to follow.

If you want to do deep dive into generating synthetic data, you can review our Synthetic Data Generation Cookbook [here](https://cookbook.openai.com/examples/sdg1)


key: _node_content
string_value:
{"id_": "6d1faed8-58e0-47b4-b542-ac24da4fe817", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da3e88ec-111f-4abd-b194-88ac4fb52157", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a26e31f25bd0e21ae0a3e7458579456580143d2e14e7bc0792e65dbd4e3e3f02", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d", "node_type": "1", "metadata": {}, "hash": "4578abbe6ecdb750e6c7b47c0d256703667cc897656f106bd8016c7fe80d6e54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 701, "end_char_idx": 4395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from concurrent.futures import ThreadPoolExecutor
from IPython.display import display, HTML
import json
import pandas as pd
from sklearn.metrics import precision_score, recall_score
from typing import List
from openai import OpenAI

client = OpenAI()
```


```python
# Function to set up display options for pandas
def setup_pandas_display():
    # Increase display limits
    pd.set_option('display.max_rows', 500)
    pd.set_option('display.max_columns', 500)

# Function to make DataFrame scrollable in the notebook output
def make_scrollable(df):
    style = (
        '<style>'
        'div.output_scroll {'
        'resize: both;'
        'overflow: auto;'
        '}'
        '</style>'
    )
    html = f"{style}{df.to_html()}"
    display(HTML(html))

# Main function to display DataFrame
def display_dataframe(df):
    setup_pandas_display()    # Enable scrollable view
    make_scrollable(df)

```

## 1. Building out an eval set

Imagine we are a customer support team that is building out an automated support agent. We will be feeding the assistant information from our knowledge base about a specific set of policies for how to handle tickets such as returns, refunds, feedback, and expect the model to follow the policy when interacting with customers.

The first thing we will do is use GPT-4o to build out a set of policies that we will want to follow.

If you want to do deep dive into generating synthetic data, you can review our Synthetic Data Generation Cookbook [here](https://cookbook.openai.com/examples/sdg1)


```python
system_input_prompt = """
You are a helpful assistant that can generate policies for a support agent at a fictional company to follow. You will be provided with a topic (ie. returns, refunds, feedback) and you are to generate a sample policy for how to handle the it.

When constructing the policy, it should contain step-by-step instructions for how to handle the customer inquiry. It should include decision logic for what to do if a customer falls under a certain category, and provide requirements for taking specific actions.
"""

user_policy_example_1 = """"
RETURN POLICY
"""

assistant_policy_example_1 = """
RETURN POLICY

1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.

"""

user_policy_input = """
{{POLICY}}
"""
```


key: _node_content
string_value:
{"id_": "24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d1faed8-58e0-47b4-b542-ac24da4fe817", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1bd3cdf139ddbd854e9058a7d2cf6b60ab6604e62bd3f2e77996ce51407510c5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "929580a0-8b53-4fb0-b821-840dfa383685", "node_type": "1", "metadata": {}, "hash": "3beb5ea763c30e3caf60e5016eb2bbff7e876cae17efc82e16979d4c0e743002", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4398, "end_char_idx": 7715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def generate_policy(policy: str) -> str:
    input_message = user_policy_input.replace("{{POLICY}}", policy)
    
    response = client.chat.completions.create(
        messages= [
            {"role": "system", "content": system_input_prompt},
            {"role": "user", "content": user_policy_example_1},
            {"role": "assistant", "content": assistant_policy_example_1},
            {"role": "user", "content": input_message},
        ],
        model="gpt-4o"
    )
    
    return response.choices[0].message.content

def generate_policies() -> List[str]:
    # List of different types of policies to generate 
    policies = ['PRODUCT FEEDBACK POLICY', 'SHIPPING POLICY', 'WARRANTY POLICY', 'ACCOUNT DELETION', 'COMPLAINT RESOLUTION']
    
    with ThreadPoolExecutor() as executor:
        policy_instructions_list = list(executor.map(generate_policy, policies))
        
    return policy_instructions_list

policy_instructions = generate_policies()
```

Next we'll take these policies and generate sample customer interactions that do or do not follow the instructions.


```python
system_input_prompt = """"
You are a helpful assistant that can generate fictional interactions between a support assistant and a customer user. You will be given a set of policy instructions that the support agent is instructed to follow.

Based on the instructions, you must generate a relevant single-turn or multi-turn interaction between the assistant and the user. It should average between 1-3 turns total.

For a given set of instructions, generate an example conversation that where the assistant either does or does not follow the instructions properly. In the assistant's responses, have it give a combination of single sentence and multi-sentence responses.

The output must be in a json format with the following three parameters:
 - accurate: 
    - This should be a boolean True or False value that matches whether or not the final assistant message accurately follows the policy instructions
 - kb_article:
    - This should be the entire policy instruction that is passed in from the user
 - chat_history: 
    - This should contain the entire conversation history except for the final assistant message. 
    - This should be in a format of an array of jsons where each json contains two parameters: role, and content. 
    - Role should be set to either 'user' to represent the customer, or 'assistant' to represent the customer support assistant. 
    - Content should contain the message from the appropriate role.
    - The final message in the chat history should always come from the user. The assistant response in the following parameter will be a response to this use message.
 - assistant_response: 
    - This should contain the final response from the assistant. This is what we will evaluate to determine whether or not it is accurately following the policy.
"""

user_example_1 = """"
Here are the policy instructions:
RETURN POLICY

1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a.


key: _node_content
string_value:
{"id_": "929580a0-8b53-4fb0-b821-840dfa383685", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c0a830da3279b08b20aa569ee4c3d32da546d9a95b9cef715fb83b07f8963f5f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad05a7c6-5d23-404e-9192-b3a3fef8411d", "node_type": "1", "metadata": {}, "hash": "26edfa70bcb2c4b56abe23aff655525e63558171ae2c415da5643636301ba1f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5749, "end_char_idx": 9219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Based on the instructions, you must generate a relevant single-turn or multi-turn interaction between the assistant and the user. It should average between 1-3 turns total.

For a given set of instructions, generate an example conversation that where the assistant either does or does not follow the instructions properly. In the assistant's responses, have it give a combination of single sentence and multi-sentence responses.

The output must be in a json format with the following three parameters:
 - accurate: 
    - This should be a boolean True or False value that matches whether or not the final assistant message accurately follows the policy instructions
 - kb_article:
    - This should be the entire policy instruction that is passed in from the user
 - chat_history: 
    - This should contain the entire conversation history except for the final assistant message. 
    - This should be in a format of an array of jsons where each json contains two parameters: role, and content. 
    - Role should be set to either 'user' to represent the customer, or 'assistant' to represent the customer support assistant. 
    - Content should contain the message from the appropriate role.
    - The final message in the chat history should always come from the user. The assistant response in the following parameter will be a response to this use message.
 - assistant_response: 
    - This should contain the final response from the assistant. This is what we will evaluate to determine whether or not it is accurately following the policy.
"""

user_example_1 = """"
Here are the policy instructions:
RETURN POLICY

1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
"""

assistant_example_1 = """
{
    "accurate": "true",
    "kb_article": "1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a.


key: _node_content
string_value:
{"id_": "ad05a7c6-5d23-404e-9192-b3a3fef8411d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "929580a0-8b53-4fb0-b821-840dfa383685", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d80fe7d800f23f75fab9aefdae054184fd588205119338363c3529440f14338d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "199e0a25-3473-4afe-85e5-8d59f324e75e", "node_type": "1", "metadata": {}, "hash": "b668c2cc8bdba5848c31aad95d8b1964ef1be653197e18250518d23252936e18", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7668, "end_char_idx": 10927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
"""

assistant_example_1 = """
{
    "accurate": "true",
    "kb_article": "1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.",
    "chat_history": [
        {
            "role": "user",
            "content": "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content": "Yes, I am not satisfied with the design"
        }
    ],
    "assistant_response": {
        "role": "assistant",
        "content": "I see. Because the shirt was ordered in the last 30 days, we can provide you with a full refund.


key: _node_content
string_value:
{"id_": "199e0a25-3473-4afe-85e5-8d59f324e75e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad05a7c6-5d23-404e-9192-b3a3fef8411d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ac004b60a562d7ef8e9a8c3e0adb5a978335b4651ace88f711b73f18be39844a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f", "node_type": "1", "metadata": {}, "hash": "8b186bb309a874c900325b0b981e72c4219c4c26196bca2a9a491b6c4d1d9c89", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9220, "end_char_idx": 12513, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.",
    "chat_history": [
        {
            "role": "user",
            "content": "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content": "Yes, I am not satisfied with the design"
        }
    ],
    "assistant_response": {
        "role": "assistant",
        "content": "I see. Because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?"
    }
}
"""

user_example_2 = """"
Here are the policy instructions:
RETURN POLICY

1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
"""


key: _node_content
string_value:
{"id_": "0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "199e0a25-3473-4afe-85e5-8d59f324e75e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9da5485a36e09dbc1fd3f9ac8fee005a8bb0134b5f4c3794dca4f6fef26d0b5c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1cde6bea-e4a6-40e7-9192-3fe587192e66", "node_type": "1", "metadata": {}, "hash": "75032821da90bf29599370e3c9942e634d6195eb4670b9f100d635d94b96111a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10838, "end_char_idx": 13967, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?"
    }
}
"""

user_example_2 = """"
Here are the policy instructions:
RETURN POLICY

1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
"""

assistant_example_2 = """
{
    "accurate": "false",
    "kb_article": "1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.


key: _node_content
string_value:
{"id_": "1cde6bea-e4a6-40e7-9192-3fe587192e66", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "48f17b420aa1ac4304896827ed1c44f556d7e9306aacce59bbf9f9db15f56b77", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "343c2a09-cb42-4afa-b73e-a3134f9c7c08", "node_type": "1", "metadata": {}, "hash": "309dc015d38a987af30a8a46ebab11f778f8f9f10b960f6ad499390920864a3f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12515, "end_char_idx": 15759, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
assistant_example_2 = """
{
    "accurate": "false",
    "kb_article": "1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.",
    "chat_history": [
        {
            "role": "user",
            "content": "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content": "Yes, I am not satisfied with the design"
        }
    ],
    "assistant_response": {
        "role": "assistant",
        "content": "I see. Because the shirt was ordered in the last 60 days, we cannot process a refund."    
    }
}
"""
```

Now let's iterate through the policies and generate some examples.


```python
customer_interactions = []

def fetch_response(policy):
    messages = [
        { "role": "system", "content": system_input_prompt},
        { "role": "user", "content": user_example_1},
        { "role": "assistant", "content": assistant_example_1},
        { "role": "user", "content": user_example_2},
        { "role": "assistant", "content": assistant_example_2},
        { "role": "user", "content": policy}
    ]

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=0.7,
        n=10
    )
    return response.choices

with ThreadPoolExecutor() as executor:
    futures = [executor.submit(fetch_response, policy) for policy in policy_instructions]
    for future in futures:
        choices = future.result()
        customer_interactions.extend([choice.message.content for choice in choices])

```


```python
interaction_dict = json.loads(customer_interactions[0])

df_interaction = pd.DataFrame([interaction_dict])

# Pretty print the DataFrame
display_dataframe(df_interaction)

```


key: _node_content
string_value:
{"id_": "343c2a09-cb42-4afa-b73e-a3134f9c7c08", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1cde6bea-e4a6-40e7-9192-3fe587192e66", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dba06ca708d2f356175f4c43d7c62870a859c40463dfa89486e65ee44c8d0428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f", "node_type": "1", "metadata": {}, "hash": "f4795af09643da5c7374ca1bc0fe375d92b2ea8ea151ae1cc9c486a2fc099f75", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14224, "end_char_idx": 19157, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content": "Yes, I am not satisfied with the design"
        }
    ],
    "assistant_response": {
        "role": "assistant",
        "content": "I see. Because the shirt was ordered in the last 60 days, we cannot process a refund."    
    }
}
"""
```

Now let's iterate through the policies and generate some examples.


```python
customer_interactions = []

def fetch_response(policy):
    messages = [
        { "role": "system", "content": system_input_prompt},
        { "role": "user", "content": user_example_1},
        { "role": "assistant", "content": assistant_example_1},
        { "role": "user", "content": user_example_2},
        { "role": "assistant", "content": assistant_example_2},
        { "role": "user", "content": policy}
    ]

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=0.7,
        n=10
    )
    return response.choices

with ThreadPoolExecutor() as executor:
    futures = [executor.submit(fetch_response, policy) for policy in policy_instructions]
    for future in futures:
        choices = future.result()
        customer_interactions.extend([choice.message.content for choice in choices])

```


```python
interaction_dict = json.loads(customer_interactions[0])

df_interaction = pd.DataFrame([interaction_dict])

# Pretty print the DataFrame
display_dataframe(df_interaction)

```


<style>div.output_scroll {resize: both;overflow: auto;}</style><table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>accurate</th>
      <th>kb_article</th>
      <th>chat_history</th>
      <th>assistant_response</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Reception** - Thank the customer for taking the time to provide feedback. - Use a personalized greeting: "Thank you for your feedback, [Customer Name]. We appreciate your input." 2. **Categorize Feedback** - Determine the type of feedback: - **Positive Feedback** - **Negative Feedback** - **Suggestions for Improvement** - Document the feedback under the appropriate category in the internal database. 3. **Responding to Positive Feedback** - Express gratitude: "We're thrilled to hear that you enjoyed our product. Thank you for letting us know!" - If possible, offer a small token of appreciation (e.g., discount or voucher for future purchases). 4. **Responding to Negative Feedback** - Apologize sincerely and acknowledge the customer's concerns: "We apologize that our product did not meet your expectations. Your feedback is important to us." - Ask for additional details if necessary to understand the issue better. - Reassure the customer that their feedback will be escalated to the product development team. 5. **Responding to Suggestions** - Acknowledge the suggestion: "Thank you for your suggestion. We value input from our customers as it helps us improve our products." - Inform the customer that their suggestion will be reviewed: "We will share your idea with our product team for further consideration." 6. **Internal Processing** - Log all feedback under the respective category in the internal database. - Forward detailed feedback to the product development team bi-weekly. - High-priority issues should be escalated immediately to the senior management team. 7. **Follow-Up** - Monitor whether the customer's feedback leads to any product updates or changes. - If the customer’s feedback resulted in product enhancement, send a follow-up email to inform them: "Thank you for your valuable feedback. We wanted to let you know that we've made some improvements based on your input." 8. **Closing the Loop** - Ask if there is anything else you can assist the customer with: "Is there anything else we can help you with today?" - Close the ticket once all queries and feedback are appropriately addressed. 9. **Continuous Improvement** - Analyze feedback trends monthly to identify recurring issues and areas for improvement. - Use feedback insights for product development meetings and strategic planning sessions. By following these steps, we ensure that customer feedback is valued, documented, and acted upon to continuously improve our product offerings.</td>
      <td>[{'role': 'user', 'content': 'I wanted to let you know that the new app update is fantastic! The interface is so much smoother now.'}]</td>
      <td>{'role': 'assistant', 'content': 'Thank you for your feedback! We appreciate your input. We're thrilled to hear that you enjoyed our product. Thank you for letting us know! As a token of our appreciation, we're offering you a 10% discount on your next purchase. Is there anything else we can help you with today?'}</td>
    </tr>
  </tbody>
</table>


key: _node_content
string_value:
{"id_": "2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "343c2a09-cb42-4afa-b73e-a3134f9c7c08", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "20cc2cd7669c18df8ea9afd7294a83976ce068b9e67770280472c2fa3a2e6640", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78817c7b-9996-4088-b1f2-e46d1d25cff6", "node_type": "1", "metadata": {}, "hash": "103dc8a23e2ee00b0b14a98c04cecf17389615d68a018cda1f02799e58e4423a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19161, "end_char_idx": 21428, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Decode the JSON strings
data = [json.loads(entry) for entry in customer_interactions]

# Create a DataFrame from the cleaned data
df = pd.DataFrame(data)
```


```python
df.head(10)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>accurate</th>
      <th>kb_article</th>
      <th>chat_history</th>
      <th>assistant_response</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to give...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY\n\n1. **Acknowledge Re...</td>
      <td>[{'role': 'user', 'content': 'I really enjoyed...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to give...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1.


key: _node_content
string_value:
{"id_": "78817c7b-9996-4088-b1f2-e46d1d25cff6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e4883933195e372443bd445d0bd13ca9153ba0d30c4c008122e2074b7665eda3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "069ff022-261b-4d49-8d44-583273218e69", "node_type": "1", "metadata": {}, "hash": "93e28ecc8ca9bdaeb2a844e0080f7fb55570b43d03b3eb5399a53613fc1adba8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20425, "end_char_idx": 22376, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to give...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY\n\n1. **Acknowledge Re...</td>
      <td>[{'role': 'user', 'content': 'I really enjoyed...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to give...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I didn't like th...</td>
      <td>{'role': 'assistant', 'content': 'We apologize...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I have some feed...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>8</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I really love th...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>9</th>
      <td>true</td>
      <td>1. **Acknowledge Reception** - Thank the custo...</td>
      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "069ff022-261b-4d49-8d44-583273218e69", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78817c7b-9996-4088-b1f2-e46d1d25cff6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2d6e835d2133a001a20bc140cce537293370d783088e01ce57175442a19df98f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba11be3a-91ba-421d-88d3-840bd82d2521", "node_type": "1", "metadata": {}, "hash": "d3925001a23a585f491d557a768b8a3d7c85968af28c54962ce047131f5d4aba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21429, "end_char_idx": 24579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I didn't like th...</td>
      <td>{'role': 'assistant', 'content': 'We apologize...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I have some feed...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>8</th>
      <td>true</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I really love th...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>9</th>
      <td>true</td>
      <td>1. **Acknowledge Reception** - Thank the custo...</td>
      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
  </tbody>
</table>
</div>



## 2. Constructing our hallucination guardrail

When building out our hallucination guardrail, here are some guiding principles:

1. Provide very descriptive metrics to evaluate whether a response is accurate
- It is important to break down this idea of "truth" in easily identifiable metrics that we can measure
- Metrics like truthfulness and relevance are difficult to measure. Giving concrete ways to score the statement can result in a more accurate guardrail
2. Ensure consistency across key terminology
- It is important to keep relevant terms such as knowledge base articles, assistants, and users consistent across the prompt
- If we begin to use phrases such as assistant vs agent, the model could get confused
3. Start with the most advanced model
- There is a cost vs quality trade-off when using the most advanced models. Although GPT-4o may be more expensive, it is important to start with the most advanced model so we can ensure a high degree of accuracy
- Once we have thoroughly tested out the guardrail and are confident in its performance, we can look to reducing cost by tuning it down to gpt-3.5-turbo
4. Evaluate each sentence independently and the entire response as a whole
- If the agent returns a long response, it can be useful to break down the response to individual sentences and evaluate them independently
- In addition to that, evaluating the whole intent of the message as a whole can ensure that you don't lose important context

With all of this in mind, let's build out a guardrail system and measure its performance.


```python
guardrail_system_message = """You are a highly specialized assistant tasked with reviewing chatbot responses to identify and flag any inaccuracies or hallucinations. For each user message, you must thoroughly analyze the response by considering:
    1. Knowledge Accuracy: Does the message accurately reflect information found in the knowledge base? Assess not only direct mentions but also contextually inferred knowledge.
    2. Relevance: Does the message directly address the user's question or statement? Check if the response logically follows the user’s last message, maintaining coherence in the conversation thread.
    3.


key: _node_content
string_value:
{"id_": "ba11be3a-91ba-421d-88d3-840bd82d2521", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "069ff022-261b-4d49-8d44-583273218e69", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "eeca0d1b3d70982cb387260d13c47ca462e7c9c29aa7f7eb05a65f75ceb95eaa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dd6e262-ccee-4e7e-a010-b0755e92344a", "node_type": "1", "metadata": {}, "hash": "0b57eb644dc0995dfa5a870db5044b3362a64f94d83443dfb13bb21cb0129f33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23938, "end_char_idx": 27242, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
guardrail_system_message = """You are a highly specialized assistant tasked with reviewing chatbot responses to identify and flag any inaccuracies or hallucinations. For each user message, you must thoroughly analyze the response by considering:
    1. Knowledge Accuracy: Does the message accurately reflect information found in the knowledge base? Assess not only direct mentions but also contextually inferred knowledge.
    2. Relevance: Does the message directly address the user's question or statement? Check if the response logically follows the user’s last message, maintaining coherence in the conversation thread.
    3. Policy Compliance: Does the message adhere to company policies? Evaluate for subtleties such as misinformation, overpromises, or logical inconsistencies. Ensure the response is polite, non-discriminatory, and practical.

To perform your task you will be given the following:
    1. Knowledge Base Articles - These are your source of truth for verifying the content of assistant messages.
    2. Chat Transcript - Provides context for the conversation between the user and the assistant.
    3. Assistant Message - The message from the assistant that needs review.

For each sentence in the assistant's most recent response, assign a score based on the following criteria:
    1. Factual Accuracy:
        - Score 1 if the sentence is factually correct and corroborated by the knowledge base.
        - Score 0 if the sentence contains factual errors or unsubstantiated claims.
    2. Relevance:
        - Score 1 if the sentence directly and specifically addresses the user's question or statement without digression.
        - Score 0 if the sentence is tangential or does not build logically on the conversation thread.
    3. Policy Compliance:
        - Score 1 if the response complies with all company policies including accuracy, ethical guidelines, and user engagement standards.
        - Score 0 if it violates any aspect of the policies, such as misinformation or inappropriate content.
    4. Contextual Coherence:
        - Score 1 if the sentence maintains or enhances the coherence of the conversation, connecting logically with preceding messages.
        - Score 0 if it disrupts the flow or context of the conversation.

Include in your response an array of JSON objects for each evaluated sentence. Each JSON object should contain:
    - `sentence`: Text of the evaluated sentence.
    - `factualAccuracy`: Score for factual correctness (0 or 1).
    - `factualReference`: If scored 1, cite the exact line(s) from the knowledge base. If scored 0, provide a rationale.
    - `relevance`: Score for relevance to the user’s question (0 or 1).
    - `policyCompliance`: Score for adherence to company policies (0 or 1).
    - `contextualCoherence`: Score for maintaining conversation coherence (0 or 1).

ALWAYS RETURN YOUR RESPONSE AS AN ARRAY OF JSONS.
"""

fs_user_1 = """

## Knowledge Base Articles: 
1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a.


key: _node_content
string_value:
{"id_": "8dd6e262-ccee-4e7e-a010-b0755e92344a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba11be3a-91ba-421d-88d3-840bd82d2521", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "73e7450752e7c24c629a0fda1734c759a0ffbda55e89a35232b9deff4a59309e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a03eb692-d487-4ddb-aed5-564be422c37a", "node_type": "1", "metadata": {}, "hash": "6705a766f37c8f979827d1fa18f56ce17febe76016151668b5ce7c4683e99d74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25461, "end_char_idx": 28698, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Relevance:
        - Score 1 if the sentence directly and specifically addresses the user's question or statement without digression.
        - Score 0 if the sentence is tangential or does not build logically on the conversation thread.
    3. Policy Compliance:
        - Score 1 if the response complies with all company policies including accuracy, ethical guidelines, and user engagement standards.
        - Score 0 if it violates any aspect of the policies, such as misinformation or inappropriate content.
    4. Contextual Coherence:
        - Score 1 if the sentence maintains or enhances the coherence of the conversation, connecting logically with preceding messages.
        - Score 0 if it disrupts the flow or context of the conversation.

Include in your response an array of JSON objects for each evaluated sentence. Each JSON object should contain:
    - `sentence`: Text of the evaluated sentence.
    - `factualAccuracy`: Score for factual correctness (0 or 1).
    - `factualReference`: If scored 1, cite the exact line(s) from the knowledge base. If scored 0, provide a rationale.
    - `relevance`: Score for relevance to the user’s question (0 or 1).
    - `policyCompliance`: Score for adherence to company policies (0 or 1).
    - `contextualCoherence`: Score for maintaining conversation coherence (0 or 1).

ALWAYS RETURN YOUR RESPONSE AS AN ARRAY OF JSONS.
"""

fs_user_1 = """

## Knowledge Base Articles: 
1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
    
## Chat Transcript:
    [
        {
            "role": "user",
            "content: "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"


key: _node_content
string_value:
{"id_": "a03eb692-d487-4ddb-aed5-564be422c37a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dd6e262-ccee-4e7e-a010-b0755e92344a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8d5c73a87b943441c3d7f86e1605693ab712399dabd6095da112a9ef4dc35cc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa", "node_type": "1", "metadata": {}, "hash": "cbeaead1da77ffdbb0b18d6c5a149cd4f6fcc3b7af1f2f87418bc32ed59ca747", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27195, "end_char_idx": 30097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
    
## Chat Transcript:
    [
        {
            "role": "user",
            "content: "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content: "Yes, I am not satisfied with the design"
        }
    ]

## Assistant Message:
I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?
"""

fs_assistant_1 = """[
    {
        "sentence": "I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund.",
        "factualAccuracy": 1,
        "factualReference": "If the order was made within 30 days, notify them that they are eligible for a full refund",
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    },
    {
        "sentence": "Would you like me to process the refund?",
        "factualAccuracy": 1,
        "factualReference": "If the order was made within 30 days, notify them that they are eligible for a full refund",
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    }
]
"""
fs_user_2 = """
## Knowledge Base Articles: 
1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a.


key: _node_content
string_value:
{"id_": "1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a03eb692-d487-4ddb-aed5-564be422c37a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ca70ee09f501ddb0beed6bcd955d4c95f6d76a5adc65abef251ab0280573cb2f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8a9097c-67b2-42b7-afb4-15b969e07443", "node_type": "1", "metadata": {}, "hash": "cd6d1a0066a4fc01c0624405a08c348df7eb89d80b4087e43037c977023f9ee6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28619, "end_char_idx": 31831, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content: "Yes, I am not satisfied with the design"
        }
    ]

## Assistant Message:
I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?
"""

fs_assistant_1 = """[
    {
        "sentence": "I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund.",
        "factualAccuracy": 1,
        "factualReference": "If the order was made within 30 days, notify them that they are eligible for a full refund",
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    },
    {
        "sentence": "Would you like me to process the refund?",
        "factualAccuracy": 1,
        "factualReference": "If the order was made within 30 days, notify them that they are eligible for a full refund",
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    }
]
"""
fs_user_2 = """
## Knowledge Base Articles: 
1. ** Ask the customer why they want the order replaced **
    - Categorize their issue into one of the following buckets:
        - damaged: They received the product in a damaged state
        - satisfaction: The customer is not satisfied with the item and does not like the product.
        - unnecessary: They no longer need the item
2a. **If return category is 'damaged'
    - Ask customer for a picture of the damaged item
    - If the item is indeed damaged, continue to step 3
    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund
    - Skip step 3 and go straight to step 4

2b. **If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
    
## Chat Transcript:
    [
        {
            "role": "user",
            "content: "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content: "Yes, I am not satisfied with the design"
        },
        {
            "role": "assistant",
            "content": "I see, because the shirt was ordered in the last 60 days, we cannot process a refund."


key: _node_content
string_value:
{"id_": "f8a9097c-67b2-42b7-afb4-15b969e07443", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "68094477cff3e2264ba25dbdb507ea4b5bc11b9037bd7ce79d24d4b943c681db", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da8fcea9-71b1-49ee-a3a6-a84d416b6f73", "node_type": "1", "metadata": {}, "hash": "ca31136485cd21ed9dee3b71622a92b1fdde6f25f29d4aa87b4a8c82e505d3a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30435, "end_char_idx": 32471, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**If return category is either 'satisfaction' or 'unnecessary'**
    - Ask the customer if they can provide feedback on the quality of the item
    - If the order was made within 30 days, notify them that they are eligible for a full refund
    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%
    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund

3. **If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
    
## Chat Transcript:
    [
        {
            "role": "user",
            "content: "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content: "Yes, I am not satisfied with the design"
        },
        {
            "role": "assistant",
            "content": "I see, because the shirt was ordered in the last 60 days, we cannot process a refund."
        }
        ]
## Assistant Message: 
I see, because the shirt was ordered in the last 60 days, we cannot process a refund.
"""

fs_assistant_2 = """'[
    {
        "sentence": "I see, because the shirt was ordered in the last 60 days, we cannot process a refund.",
        "factualAccuracy": 0,
        "knowledgeReference: "If an order was placed within 60 days, you must process a partial refund."
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    }
]"""


user_input = """
## Knowledge Base Articles
{kb_articles}

## Chat Transcript
{transcript}

## Assistant Message:
{message}
"""
```


key: _node_content
string_value:
{"id_": "da8fcea9-71b1-49ee-a3a6-a84d416b6f73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8a9097c-67b2-42b7-afb4-15b969e07443", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0cfdc715bda8b2f7b1853539ff2adaa1794a1ecda89219b0baf290f06ee4473b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8332cc8f-925c-43a2-8abf-ffa88912630a", "node_type": "1", "metadata": {}, "hash": "62ee1fa69eb8e9e68ed0d17122304396a468f519f8614cbf4feb83bae0ea6f02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30896, "end_char_idx": 34646, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**If the customer is eligible for a return or refund**
    - Ask the customer to confirm that they would like a return or refund
    - Once they confirm, process their request

4 **Provide additional support before closing out ticket**
    - Ask the customer if there is anything else you can do to help them today.
    
## Chat Transcript:
    [
        {
            "role": "user",
            "content: "I would like to return this shirt"
        },
        {
            "role": "assistant",
            "content": "Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?"
        },
        {
            "role": "user",
            "content: "Yes, I am not satisfied with the design"
        },
        {
            "role": "assistant",
            "content": "I see, because the shirt was ordered in the last 60 days, we cannot process a refund."
        }
        ]
## Assistant Message: 
I see, because the shirt was ordered in the last 60 days, we cannot process a refund.
"""

fs_assistant_2 = """'[
    {
        "sentence": "I see, because the shirt was ordered in the last 60 days, we cannot process a refund.",
        "factualAccuracy": 0,
        "knowledgeReference: "If an order was placed within 60 days, you must process a partial refund."
        "relevance": 1,
        "policyCompliance": 1,
        "contextualCoherence": 1
    }
]"""


user_input = """
## Knowledge Base Articles
{kb_articles}

## Chat Transcript
{transcript}

## Assistant Message:
{message}
"""
```


```python
hallucination_outputs = []

def validate_hallucinations(row):
    kb_articles = row['kb_article']
    chat_history = row['chat_history']
    assistant_response = row['assistant_response']
    
    user_input_filled = user_input.format(
        kb_articles=kb_articles,
        transcript=chat_history,
        message=assistant_response
    )
    
    messages = [
        { "role": "system", "content": guardrail_system_message},
        { "role": "user", "content": fs_user_1},
        { "role": "assistant", "content": fs_assistant_1},
        { "role": "user", "content": fs_user_2},
        { "role": "assistant", "content": fs_assistant_2},
        { "role": "user", "content": user_input_filled}
    ]

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        temperature=0.7,
        n=10
    )
    return response.choices

# Create an empty list to store the results
results_list = []

def process_row(row):
    choices = validate_hallucinations(row)
    response_json = choices[0].message.content 
    # Parse the response content as JSON
    response_data = json.loads(response_json)
    
    for response_item in response_data:
        # Sum up the scores of the properties
        score_sum = (
            response_item.get('factualAccuracy', 0) +
            response_item.get('relevance', 0) +
            response_item.get('policyCompliance', 0) +
            response_item.get('contextualCoherence', 0)
        )
        
        # Determine if the response item is a pass or fail
        hallucination_status = 'Pass' if score_sum == 4 else 'Fail'
        
        results_list.append({
            'accurate': row['accurate'],
            'hallucination': hallucination_status,
            'kb_article': row['kb_article'],
            'chat_history': row['chat_history'],
            'assistant_response': row['assistant_response']
        })

# Use ThreadPoolExecutor to parallelize the processing of rows
with ThreadPoolExecutor() as executor:
    executor.map(process_row, [row for index, row in df.iterrows()])

# Convert the list to a DataFrame
results_df = pd.DataFrame(results_list)
```


key: _node_content
string_value:
{"id_": "8332cc8f-925c-43a2-8abf-ffa88912630a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da8fcea9-71b1-49ee-a3a6-a84d416b6f73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "86c0718e5516c24477144f6e2b08ac043c938e3815cdff7a6fd7d87ac269273b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7be22b7a-bd30-4c5c-b938-c1e58f8bfc25", "node_type": "1", "metadata": {}, "hash": "e9be690d6b763dacf73ef1dd42eb495e2367937df20cda0dc3a6bbc8b8f5a0dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34649, "end_char_idx": 36651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>accurate</th>
      <th>hallucination</th>
      <th>kb_article</th>
      <th>chat_history</th>
      <th>assistant_response</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>true</td>
      <td>Pass</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>true</td>
      <td>Pass</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>true</td>
      <td>Pass</td>
      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>
      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>true</td>
      <td>Pass</td>
      <td>1. **Acknowledge Reception** - Thank the custo...</td>
      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>true</td>
      <td>Pass</td>
      <td>1. **Acknowledge Reception** - Thank the custo...</td>
      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>
      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>
    </tr>
  </tbody>
</table>
</div>




```python
results_df.to_csv('hallucination_results.csv', index=False)

```


key: _node_content
string_value:
{"id_": "7be22b7a-bd30-4c5c-b938-c1e58f8bfc25", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8332cc8f-925c-43a2-8abf-ffa88912630a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "869b8135da11f0bb744bde9de0612599f82738d6825e11a3e5469c9c528ef311", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36577, "end_char_idx": 39074, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results_df.to_csv('hallucination_results.csv', index=False)

```


```python
df = pd.read_csv('hallucination_results.csv')

if 'accurate' not in df.columns or 'hallucination' not in df.columns:
    print("Error: The required columns are not present in the DataFrame.")
else:
    # Transform values to binary 0/1
    try:
        df['accurate'] = df['accurate'].astype(str).str.strip().map(lambda x: 1 if x in ['True', 'true'] else 0)
        df['hallucination'] = df['hallucination'].str.strip().map(lambda x: 1 if x == 'Pass' else 0)
        
    except KeyError as e:
        print(f"Mapping error: {e}")

    # Check for any NaN values after mapping
    if df['accurate'].isnull().any() or df['hallucination'].isnull().any():
        print("Error: There are NaN values in the mapped columns. Check the input data for unexpected values.")
    else:
        # Calculate precision and recall
        try:
            # Precision measures the proportion of correctly identified true positives out of all instances predicted as positive. 
            # Precision = (True Positives) / (True Positives + False Positives)
            
            precision = precision_score(df['accurate'], df['hallucination'])
            
            # Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.
            # Recall = (True Positives) / (True Positives + False Negatives)
            
            recall = recall_score(df['accurate'], df['hallucination'])
            
            
            print(f"\nPrecision: {precision:.2f} (Precision measures the proportion of correctly identified true positives out of all instances predicted as positive.), "
                  f"\nRecall: {recall:.2f} (Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.)")

        except ValueError as e:
            print(f"Error in calculating precision and recall: {e}")
```

    
    Precision: 0.97 (Precision measures the proportion of correctly identified true positives out of all instances predicted as positive.), 
    Recall: 1.00 (Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.)


From the results above we can see the program is performing well with a high precision and recall metric. This means that the guardrails are able to accurately identify hallucinations in the model outputs.


key: _node_content
string_value:
{"id_": "913de36b-d722-44ce-b1c0-b110e6609aa8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c57a679-2f0e-4b78-9d24-0952df264da1", "node_type": "1", "metadata": {}, "hash": "798d28dd6ec1e1318a81f87696398bf8e020b2a1926af9c6fe164db344ecc822", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Customizing embeddings

This notebook demonstrates one way to customize OpenAI embeddings to a particular task.

The input is training data in the form of [text_1, text_2, label] where label is +1 if the pairs are similar and -1 if the pairs are dissimilar.

The output is a matrix that you can use to multiply your embeddings. The product of this multiplication is a 'custom embedding' that will better emphasize aspects of the text relevant to your use case. In binary classification use cases, we've seen error rates drop by as much as 50%.

In the following example, I use 1,000 sentence pairs picked from the SNLI corpus. Each pair of sentences are logically entailed (i.e., one implies the other). These pairs are our positives (label = 1). We generate synthetic negatives by combining sentences from different pairs, which are presumed to not be logically entailed (label = -1).

For a clustering use case, you can generate positives by creating pairs from texts in the same clusters and generate negatives by creating pairs from sentences in different clusters.

With other data sets, we have seen decent improvement with as little as ~100 training examples. Of course, performance will be better with  more examples.

# 0. Imports


```python
# imports
from typing import List, Tuple  # for type hints

import numpy as np  # for manipulating arrays
import pandas as pd  # for manipulating data in dataframes
import pickle  # for saving the embeddings cache
import plotly.express as px  # for plots
import random  # for generating run IDs
from sklearn.model_selection import train_test_split  # for splitting train & test data
import torch  # for matrix optimization

from utils.embeddings_utils import get_embedding, cosine_similarity  # for embeddings

```

## 1. Inputs

Most inputs are here. The key things to change are where to load your datset from, where to save a cache of embeddings to, and which embedding engine you want to use.

Depending on how your data is formatted, you'll want to rewrite the process_input_data function.


```python
# input parameters
embedding_cache_path = "data/snli_embedding_cache.pkl"  # embeddings will be saved/loaded here
default_embedding_engine = "text-embedding-3-small"
num_pairs_to_embed = 1000  # 1000 is arbitrary
local_dataset_path = "data/snli_1.0_train_2k.csv"  # download from: https://nlp.stanford.edu/projects/snli/


def process_input_data(df: pd.DataFrame) -> pd.DataFrame:
    # you can customize this to preprocess your own dataset
    # output should be a dataframe with 3 columns: text_1, text_2, label (1 for similar, -1 for dissimilar)
    df["label"] = df["gold_label"]
    df = df[df["label"].isin(["entailment"])]
    df["label"] = df["label"].apply(lambda x: {"entailment": 1, "contradiction": -1}[x])
    df = df.rename(columns={"sentence1": "text_1", "sentence2": "text_2"})
    df = df[["text_1", "text_2", "label"]]
    df = df.head(num_pairs_to_embed)
    return df

```

## 2. Load and process input data


