# embedding_metadata

key: _node_content
string_value:
{"id_": "e0788c8e-37bf-48df-bc9f-124bb1a23510", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b023c0e-0ef9-409b-9c41-cb57cb5f629f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "89651e94531b6bf7e41f04e35b85fbd422e53ecc42172773a41fd7e7dba529d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e76f4265-0d8f-4b7b-ad58-77059b21bad3", "node_type": "1", "metadata": {}, "hash": "f101fd8fea7509babdac6c13e1b9fe2f6bfde5bcba7cd251848b13cd3ab62a09", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9040, "end_char_idx": 14366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Headline: Tennis Champion Showcases Hidden Talents in Symphony Orchestra Debut



<span style='color: cyan'>Output token 1:</span> Art, <span style='color: darkorange'>logprobs:</span> -0.009169078, <span style='color: magenta'>linear probability:</span> 99.09%<br><span style='color: cyan'>Output token 2:</span> Sports, <span style='color: darkorange'>logprobs:</span> -4.696669, <span style='color: magenta'>linear probability:</span> 0.91%<br>


    
    


As expected from the first two headlines, `gpt-4` is nearly 100% confident in its classifications, as the content is clearly technology and politics focused respectively. However, the third headline combines both sports and art-related themes, so we see the model is less confident in its selection.

This shows how important using `logprobs` can be, as if we are using LLMs for classification tasks we can set confidence theshholds, or output several potential output tokens if the log probability of the selected output is not sufficiently high. For instance, if we are creating a recommendation engine to tag articles, we can automatically classify headlines crossing a certain threshold, and send the less certain headlines for manual review.

## 2. Retrieval confidence scoring to reduce hallucinations

To reduce hallucinations, and the performance of our RAG-based Q&A system, we can use `logprobs` to evaluate how confident the model is in its retrieval.

Let's say we have built a retrieval system using RAG for Q&A, but are struggling with hallucinated answers to our questions. *Note:* we will use a hardcoded article for this example, but see other entries in the cookbook for tutorials on using RAG for Q&A.


```python
# Article retrieved
ada_lovelace_article = """Augusta Ada King, Countess of Lovelace (née Byron; 10 December 1815 – 27 November 1852) was an English mathematician and writer, chiefly known for her work on Charles Babbage's proposed mechanical general-purpose computer, the Analytical Engine. She was the first to recognise that the machine had applications beyond pure calculation.
Ada Byron was the only legitimate child of poet Lord Byron and reformer Lady Byron. All Lovelace's half-siblings, Lord Byron's other children, were born out of wedlock to other women. Byron separated from his wife a month after Ada was born and left England forever. He died in Greece when Ada was eight. Her mother was anxious about her upbringing and promoted Ada's interest in mathematics and logic in an effort to prevent her from developing her father's perceived insanity. Despite this, Ada remained interested in him, naming her two sons Byron and Gordon. Upon her death, she was buried next to him at her request. Although often ill in her childhood, Ada pursued her studies assiduously. She married William King in 1835. King was made Earl of Lovelace in 1838, Ada thereby becoming Countess of Lovelace.
Her educational and social exploits brought her into contact with scientists such as Andrew Crosse, Charles Babbage, Sir David Brewster, Charles Wheatstone, Michael Faraday, and the author Charles Dickens, contacts which she used to further her education. Ada described her approach as "poetical science" and herself as an "Analyst (& Metaphysician)".
When she was eighteen, her mathematical talents led her to a long working relationship and friendship with fellow British mathematician Charles Babbage, who is known as "the father of computers". She was in particular interested in Babbage's work on the Analytical Engine. Lovelace first met him in June 1833, through their mutual friend, and her private tutor, Mary Somerville.
Between 1842 and 1843, Ada translated an article by the military engineer Luigi Menabrea (later Prime Minister of Italy) about the Analytical Engine, supplementing it with an elaborate set of seven notes, simply called "Notes".
Lovelace's notes are important in the early history of computers, especially since the seventh one contained what many consider to be the first computer program—that is, an algorithm designed to be carried out by a machine. Other historians reject this perspective and point out that Babbage's personal notes from the years 1836/1837 contain the first programs for the engine. She also developed a vision of the capability of computers to go beyond mere calculating or number-crunching, while many others, including Babbage himself, focused only on those capabilities. Her mindset of "poetical science" led her to ask questions about the Analytical Engine (as shown in her notes) examining how individuals and society relate to technology as a collaborative tool.
"""

# Questions that can be easily answered given the article
easy_questions = [
    "What nationality was Ada Lovelace?",
    "What was an important finding from Lovelace's seventh note?",
]

# Questions that are not fully covered in the article
medium_questions = [
    "Did Lovelace collaborate with Charles Dickens",
    "What concepts did Lovelace build with Charles Babbage",
]
```

Now, what we can do is ask the model to respond to the question, but then also evaluate its response. Specifically, we will ask the model to output a boolean `has_sufficient_context_for_answer`. We can then evaluate the `logprobs` to see just how confident the model is that its answer was contained in the provided context


key: _node_content
string_value:
{"id_": "e76f4265-0d8f-4b7b-ad58-77059b21bad3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0788c8e-37bf-48df-bc9f-124bb1a23510", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "faf55353813466824aa9706ac6b88bfd423de73974e636f764eecf244692e1e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45919a2d-0240-43a3-81ae-ce84a59b0f72", "node_type": "1", "metadata": {}, "hash": "d962395f75158624b821cd60a0b7c486fb99c8cdef821cd601b0307e73353bc9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14369, "end_char_idx": 16505, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
PROMPT = """You retrieved this article: {article}. The question is: {question}.
Before even answering the question, consider whether you have sufficient information in the article to answer the question fully.
Your output should JUST be the boolean true or false, of if you have sufficient information in the article to answer the question.
Respond with just one word, the boolean true or false. You must output the word 'True', or the word 'False', nothing else.
"""

```


```python
html_output = ""
html_output += "Questions clearly answered in article"

for question in easy_questions:
    API_RESPONSE = get_completion(
        [
            {
                "role": "user",
                "content": PROMPT.format(
                    article=ada_lovelace_article, question=question
                ),
            }
        ],
        model="gpt-4",
        logprobs=True,
    )
    html_output += f'<p style="color:green">Question: {question}</p>'
    for logprob in API_RESPONSE.choices[0].logprobs.content:
        html_output += f'<p style="color:cyan">has_sufficient_context_for_answer: {logprob.token}, <span style="color:darkorange">logprobs: {logprob.logprob}, <span style="color:magenta">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'

html_output += "Questions only partially covered in the article"

for question in medium_questions:
    API_RESPONSE = get_completion(
        [
            {
                "role": "user",
                "content": PROMPT.format(
                    article=ada_lovelace_article, question=question
                ),
            }
        ],
        model="gpt-4",
        logprobs=True,
        top_logprobs=3,
    )
    html_output += f'<p style="color:green">Question: {question}</p>'
    for logprob in API_RESPONSE.choices[0].logprobs.content:
        html_output += f'<p style="color:cyan">has_sufficient_context_for_answer: {logprob.token}, <span style="color:darkorange">logprobs: {logprob.logprob}, <span style="color:magenta">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'

display(HTML(html_output))
```


key: _node_content
string_value:
{"id_": "45919a2d-0240-43a3-81ae-ce84a59b0f72", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e76f4265-0d8f-4b7b-ad58-77059b21bad3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6ce31c81369dd003f49a35dc6e29b56e4ae3799e430648448b54e81b9898dd4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3e196cc-2e6c-4304-94c4-123bb41ab272", "node_type": "1", "metadata": {}, "hash": "3b35b3c7ba36a28a7aee8e8375d4a560db7cf7c8935f4cc170895302eef33040", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14854, "end_char_idx": 17663, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
html_output = ""
html_output += "Questions clearly answered in article"

for question in easy_questions:
    API_RESPONSE = get_completion(
        [
            {
                "role": "user",
                "content": PROMPT.format(
                    article=ada_lovelace_article, question=question
                ),
            }
        ],
        model="gpt-4",
        logprobs=True,
    )
    html_output += f'<p style="color:green">Question: {question}</p>'
    for logprob in API_RESPONSE.choices[0].logprobs.content:
        html_output += f'<p style="color:cyan">has_sufficient_context_for_answer: {logprob.token}, <span style="color:darkorange">logprobs: {logprob.logprob}, <span style="color:magenta">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'

html_output += "Questions only partially covered in the article"

for question in medium_questions:
    API_RESPONSE = get_completion(
        [
            {
                "role": "user",
                "content": PROMPT.format(
                    article=ada_lovelace_article, question=question
                ),
            }
        ],
        model="gpt-4",
        logprobs=True,
        top_logprobs=3,
    )
    html_output += f'<p style="color:green">Question: {question}</p>'
    for logprob in API_RESPONSE.choices[0].logprobs.content:
        html_output += f'<p style="color:cyan">has_sufficient_context_for_answer: {logprob.token}, <span style="color:darkorange">logprobs: {logprob.logprob}, <span style="color:magenta">linear probability: {np.round(np.exp(logprob.logprob)*100,2)}%</span></p>'

display(HTML(html_output))
```


Questions clearly answered in article<p style="color:green">Question: What nationality was Ada Lovelace?</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -3.1281633e-07, <span style="color:magenta">linear probability: 100.0%</span></p><p style="color:green">Question: What was an important finding from Lovelace's seventh note?</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -7.89631e-07, <span style="color:magenta">linear probability: 100.0%</span></p>Questions only partially covered in the article<p style="color:green">Question: Did Lovelace collaborate with Charles Dickens</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -0.06993677, <span style="color:magenta">linear probability: 93.25%</span></p><p style="color:green">Question: What concepts did Lovelace build with Charles Babbage</p><p style="color:cyan">has_sufficient_context_for_answer: False, <span style="color:darkorange">logprobs: -0.61807257, <span style="color:magenta">linear probability: 53.9%</span></p>


key: _node_content
string_value:
{"id_": "e3e196cc-2e6c-4304-94c4-123bb41ab272", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45919a2d-0240-43a3-81ae-ce84a59b0f72", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "70c489fe666f9cca537b384a457728c9a0c2aba92571b3d53b354bd38d07e7b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2cfbfceb-74ff-4a60-88dc-945ba5682f18", "node_type": "1", "metadata": {}, "hash": "17eb89701ff59b70af19a91a422f91e6a599cd5b8339b4e30e002ac0769d8acb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16508, "end_char_idx": 19483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Questions clearly answered in article<p style="color:green">Question: What nationality was Ada Lovelace?</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -3.1281633e-07, <span style="color:magenta">linear probability: 100.0%</span></p><p style="color:green">Question: What was an important finding from Lovelace's seventh note?</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -7.89631e-07, <span style="color:magenta">linear probability: 100.0%</span></p>Questions only partially covered in the article<p style="color:green">Question: Did Lovelace collaborate with Charles Dickens</p><p style="color:cyan">has_sufficient_context_for_answer: True, <span style="color:darkorange">logprobs: -0.06993677, <span style="color:magenta">linear probability: 93.25%</span></p><p style="color:green">Question: What concepts did Lovelace build with Charles Babbage</p><p style="color:cyan">has_sufficient_context_for_answer: False, <span style="color:darkorange">logprobs: -0.61807257, <span style="color:magenta">linear probability: 53.9%</span></p>


For the first two questions, our model asserts with (near) 100% confidence that the article has sufficient context to answer the posed questions.<br><br>
On the other hand, for the more tricky questions which are less clearly answered in the article, the model is less confident that it has sufficient context. This is a great guardrail to help ensure our retrieved content is sufficient.<br><br>
This self-evaluation can help reduce hallucinations, as you can restrict answers or re-prompt the user when your `sufficient_context_for_answer` log probability is below a certain threshold. Methods like this have been shown to significantly reduce RAG for Q&A hallucinations and errors ([Example](https://jfan001.medium.com/how-we-cut-the-rate-of-gpt-hallucinations-from-20-to-less-than-2-f3bfcc10e4ec)) 


```python

```

## 3. Autocomplete

Another use case for `logprobs` are autocomplete systems. Without creating the entire autocomplete system end-to-end, let's demonstrate how `logprobs` could help us decide how to suggest words as a user is typing.

First, let's come up with a sample sentence: `"My least favorite TV show is Breaking Bad."` Let's say we want it to dynamically recommend the next word or token as we are typing the sentence, but *only* if the model is quite sure of what the next word will be. To demonstrate this, let's break up the sentence into sequential components.


```python
sentence_list = [
    "My",
    "My least",
    "My least favorite",
    "My least favorite TV",
    "My least favorite TV show",
    "My least favorite TV show is",
    "My least favorite TV show is Breaking Bad",
]
```

Now, we can ask `gpt-3.5-turbo` to act as an autocomplete engine with whatever context the model is given. We can enable `logprobs` and can see how confident the model is in its prediction.


key: _node_content
string_value:
{"id_": "2cfbfceb-74ff-4a60-88dc-945ba5682f18", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3e196cc-2e6c-4304-94c4-123bb41ab272", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3f565ca1d81e4a8e8a577eb4faa5856ab671762ecd41852003a0f0cfc9881bb6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86269357-d6d5-4e49-abe6-7eba8a95907b", "node_type": "1", "metadata": {}, "hash": "3b2aced285b3fb80afce474dc95c633dae1c447e3bc2e38fbedf07ed4fd7eb88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18471, "end_char_idx": 21345, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python

```

## 3. Autocomplete

Another use case for `logprobs` are autocomplete systems. Without creating the entire autocomplete system end-to-end, let's demonstrate how `logprobs` could help us decide how to suggest words as a user is typing.

First, let's come up with a sample sentence: `"My least favorite TV show is Breaking Bad."` Let's say we want it to dynamically recommend the next word or token as we are typing the sentence, but *only* if the model is quite sure of what the next word will be. To demonstrate this, let's break up the sentence into sequential components.


```python
sentence_list = [
    "My",
    "My least",
    "My least favorite",
    "My least favorite TV",
    "My least favorite TV show",
    "My least favorite TV show is",
    "My least favorite TV show is Breaking Bad",
]
```

Now, we can ask `gpt-3.5-turbo` to act as an autocomplete engine with whatever context the model is given. We can enable `logprobs` and can see how confident the model is in its prediction.


```python
high_prob_completions = {}
low_prob_completions = {}
html_output = ""

for sentence in sentence_list:
    PROMPT = """Complete this sentence. You are acting as auto-complete. Simply complete the sentence to the best of your ability, make sure it is just ONE sentence: {sentence}"""
    API_RESPONSE = get_completion(
        [{"role": "user", "content": PROMPT.format(sentence=sentence)}],
        model="gpt-3.5-turbo",
        logprobs=True,
        top_logprobs=3,
    )
    html_output += f'<p>Sentence: {sentence}</p>'
    first_token = True
    for token in API_RESPONSE.choices[0].logprobs.content[0].top_logprobs:
        html_output += f'<p style="color:cyan">Predicted next token: {token.token}, <span style="color:darkorange">logprobs: {token.logprob}, <span style="color:magenta">linear probability: {np.round(np.exp(token.logprob)*100,2)}%</span></p>'
        if first_token:
            if np.exp(token.logprob) > 0.95:
                high_prob_completions[sentence] = token.token
            if np.exp(token.logprob) < 0.60:
                low_prob_completions[sentence] = token.token
        first_token = False
    html_output += "<br>"

display(HTML(html_output))
```


<p>Sentence: My</p><p style="color:cyan">Predicted next token: favorite, <span style="color:darkorange">logprobs: -0.18245785, <span style="color:magenta">linear probability: 83.32%</span></p><p style="color:cyan">Predicted next token: dog, <span style="color:darkorange">logprobs: -2.397172, <span style="color:magenta">linear probability: 9.1%</span></p><p style="color:cyan">Predicted next token: ap, <span style="color:darkorange">logprobs: -3.8732424, <span style="color:magenta">linear probability: 2.08%</span></p><br><p>Sentence: My least</p><p style="color:cyan">Predicted next token: favorite, <span style="color:darkorange">logprobs: -0.0146376295,


key: _node_content
string_value:
{"id_": "86269357-d6d5-4e49-abe6-7eba8a95907b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2cfbfceb-74ff-4a60-88dc-945ba5682f18", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "23e1be41f644223f654013aedca105db5a4bc67332c8ea5e3850889725e13ad4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b53ba361-1d73-4f55-94d7-b0f53b2589cf", "node_type": "1", "metadata": {}, "hash": "489c28551420e75cb21109bd85577eebad9f785bf39d9f743e6c49aa2853e28c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20686, "end_char_idx": 23091, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<p>Sentence: My</p><p style="color:cyan">Predicted next token: favorite, <span style="color:darkorange">logprobs: -0.18245785, <span style="color:magenta">linear probability: 83.32%</span></p><p style="color:cyan">Predicted next token: dog, <span style="color:darkorange">logprobs: -2.397172, <span style="color:magenta">linear probability: 9.1%</span></p><p style="color:cyan">Predicted next token: ap, <span style="color:darkorange">logprobs: -3.8732424, <span style="color:magenta">linear probability: 2.08%</span></p><br><p>Sentence: My least</p><p style="color:cyan">Predicted next token: favorite, <span style="color:darkorange">logprobs: -0.0146376295, <span style="color:magenta">linear probability: 98.55%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -4.2417912, <span style="color:magenta">linear probability: 1.44%</span></p><p style="color:cyan">Predicted next token:  favorite, <span style="color:darkorange">logprobs: -9.748788, <span style="color:magenta">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite</p><p style="color:cyan">Predicted next token: food, <span style="color:darkorange">logprobs: -0.9481721, <span style="color:magenta">linear probability: 38.74%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -1.3447137, <span style="color:magenta">linear probability: 26.06%</span></p><p style="color:cyan">Predicted next token: color, <span style="color:darkorange">logprobs: -1.3887696, <span style="color:magenta">linear probability: 24.94%</span></p><br><p>Sentence: My least favorite TV</p><p style="color:cyan">Predicted next token: show, <span style="color:darkorange">logprobs: -0.0007898556, <span style="color:magenta">linear probability: 99.92%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -7.711523, <span style="color:magenta">linear probability: 0.04%</span></p><p style="color:cyan">Predicted next token: series, <span style="color:darkorange">logprobs: -9.348547, <span style="color:magenta">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite TV show</p><p style="color:cyan">Predicted next token: is, <span style="color:darkorange">logprobs: -0.2851253, <span style="color:magenta">linear probability: 75.19%</span></p><p style="color:cyan">Predicted next token: of,


key: _node_content
string_value:
{"id_": "b53ba361-1d73-4f55-94d7-b0f53b2589cf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86269357-d6d5-4e49-abe6-7eba8a95907b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6f134d6dc2eb5582d2ad4ef30dbd65f3dd5471be2f300b656240cc3e8c4755e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "364cf924-fd23-452f-80b9-6fa58f4cb2db", "node_type": "1", "metadata": {}, "hash": "18c3728606e296f31e9ba51726c3a99a654a83599580f7e023a8b8911119c86a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9481721, <span style="color:magenta">linear probability: 38.74%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -1.3447137, <span style="color:magenta">linear probability: 26.06%</span></p><p style="color:cyan">Predicted next token: color, <span style="color:darkorange">logprobs: -1.3887696, <span style="color:magenta">linear probability: 24.94%</span></p><br><p>Sentence: My least favorite TV</p><p style="color:cyan">Predicted next token: show, <span style="color:darkorange">logprobs: -0.0007898556, <span style="color:magenta">linear probability: 99.92%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -7.711523, <span style="color:magenta">linear probability: 0.04%</span></p><p style="color:cyan">Predicted next token: series, <span style="color:darkorange">logprobs: -9.348547, <span style="color:magenta">linear probability: 0.01%</span></p><br><p>Sentence: My least favorite TV show</p><p style="color:cyan">Predicted next token: is, <span style="color:darkorange">logprobs: -0.2851253, <span style="color:magenta">linear probability: 75.19%</span></p><p style="color:cyan">Predicted next token: of, <span style="color:darkorange">logprobs: -1.55335, <span style="color:magenta">linear probability: 21.15%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -3.4928775, <span style="color:magenta">linear probability: 3.04%</span></p><br><p>Sentence: My least favorite TV show is</p><p style="color:cyan">Predicted next token: "My, <span style="color:darkorange">logprobs: -0.69349754, <span style="color:magenta">linear probability: 49.98%</span></p><p style="color:cyan">Predicted next token: "The, <span style="color:darkorange">logprobs: -1.2899293, <span style="color:magenta">linear probability: 27.53%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -2.4170141, <span style="color:magenta">linear probability: 8.92%</span></p><br><p>Sentence: My least favorite TV show is Breaking Bad</p><p style="color:cyan">Predicted next token: because, <span style="color:darkorange">logprobs: -0.17786823, <span style="color:magenta">linear probability: 83.71%</span></p><p style="color:cyan">Predicted next token: , <span style="color:darkorange">logprobs: -2.3946173,


key: _node_content
string_value:
{"id_": "364cf924-fd23-452f-80b9-6fa58f4cb2db", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b53ba361-1d73-4f55-94d7-b0f53b2589cf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3b8c664b2f32c10d2bff5caa56df79e32fb34a3726d2320f8bcd61676f03c7a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "55212b9e-0037-4057-8a7c-c74e66111b5d", "node_type": "1", "metadata": {}, "hash": "83a39db323f2cd5410308436c5705c10bb39796273d4c59647b3164ecd485bdd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
19%</span></p><p style="color:cyan">Predicted next token: of, <span style="color:darkorange">logprobs: -1.55335, <span style="color:magenta">linear probability: 21.15%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -3.4928775, <span style="color:magenta">linear probability: 3.04%</span></p><br><p>Sentence: My least favorite TV show is</p><p style="color:cyan">Predicted next token: "My, <span style="color:darkorange">logprobs: -0.69349754, <span style="color:magenta">linear probability: 49.98%</span></p><p style="color:cyan">Predicted next token: "The, <span style="color:darkorange">logprobs: -1.2899293, <span style="color:magenta">linear probability: 27.53%</span></p><p style="color:cyan">Predicted next token: My, <span style="color:darkorange">logprobs: -2.4170141, <span style="color:magenta">linear probability: 8.92%</span></p><br><p>Sentence: My least favorite TV show is Breaking Bad</p><p style="color:cyan">Predicted next token: because, <span style="color:darkorange">logprobs: -0.17786823, <span style="color:magenta">linear probability: 83.71%</span></p><p style="color:cyan">Predicted next token: , <span style="color:darkorange">logprobs: -2.3946173, <span style="color:magenta">linear probability: 9.12%</span></p><p style="color:cyan">Predicted next token: . <span style="color:darkorange">logprobs: -3.1861975, <span style="color:magenta">linear probability: 4.13%</span></p><br>


Let's look at the high confidence autocompletions:


```python
high_prob_completions

```




    {'My least': 'favorite', 'My least favorite TV': 'show'}



These look reasonable! We can feel confident in those suggestions. It's pretty likely you want to write 'show' after writing 'My least favorite TV'! Now let's look at the autocompletion suggestions the model was less confident about:


```python
low_prob_completions

```




    {'My least favorite': 'food', 'My least favorite TV show is': '"My'}



These are logical as well. It's pretty unclear what the user is going to say with just the prefix 'my least favorite', and it's really anyone's guess what the author's favorite TV show is. <br><br>
So, using `gpt-3.5-turbo`, we can create the root of a dynamic autocompletion engine with `logprobs`!

## 4. Highlighter and bytes parameter

Let's quickly touch on creating a simple token highlighter with `logprobs`, and using the bytes parameter. First, we can create a function that counts and highlights each token. While this doesn't use the log probabilities, it uses the built in tokenization that comes with enabling `logprobs`.


key: _node_content
string_value:
{"id_": "55212b9e-0037-4057-8a7c-c74e66111b5d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "364cf924-fd23-452f-80b9-6fa58f4cb2db", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dfcfc1d4684fae02241e5f6bb237304ac5f11d9c694b1a5fc52b24f959928427", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d3ba58f-4b7c-4f4b-8fd4-2a564fd32a77", "node_type": "1", "metadata": {}, "hash": "e85ee81c864e968eed4cd8227df8ea814e333dbab864ee74d5403b5476ddb5a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
71%</span></p><p style="color:cyan">Predicted next token: , <span style="color:darkorange">logprobs: -2.3946173, <span style="color:magenta">linear probability: 9.12%</span></p><p style="color:cyan">Predicted next token: . <span style="color:darkorange">logprobs: -3.1861975, <span style="color:magenta">linear probability: 4.13%</span></p><br>


Let's look at the high confidence autocompletions:


```python
high_prob_completions

```




    {'My least': 'favorite', 'My least favorite TV': 'show'}



These look reasonable! We can feel confident in those suggestions. It's pretty likely you want to write 'show' after writing 'My least favorite TV'! Now let's look at the autocompletion suggestions the model was less confident about:


```python
low_prob_completions

```




    {'My least favorite': 'food', 'My least favorite TV show is': '"My'}



These are logical as well. It's pretty unclear what the user is going to say with just the prefix 'my least favorite', and it's really anyone's guess what the author's favorite TV show is. <br><br>
So, using `gpt-3.5-turbo`, we can create the root of a dynamic autocompletion engine with `logprobs`!

## 4. Highlighter and bytes parameter

Let's quickly touch on creating a simple token highlighter with `logprobs`, and using the bytes parameter. First, we can create a function that counts and highlights each token. While this doesn't use the log probabilities, it uses the built in tokenization that comes with enabling `logprobs`.


```python
PROMPT = """What's the longest word in the English language?"""

API_RESPONSE = get_completion(
    [{"role": "user", "content": PROMPT}], model="gpt-4", logprobs=True, top_logprobs=5
)


def highlight_text(api_response):
    colors = [
        "#FF00FF",  # Magenta
        "#008000",  # Green
        "#FF8C00",  # Dark Orange
        "#FF0000",  # Red
        "#0000FF",  # Blue
    ]
    tokens = api_response.choices[0].logprobs.content

    color_idx = 0  # Initialize color index
    html_output = ""  # Initialize HTML output
    for t in tokens:
        token_str = bytes(t.bytes).decode("utf-8")  # Decode bytes to string

        # Add colored token to HTML output
        html_output += f"<span style='color: {colors[color_idx]}'>{token_str}</span>"

        # Move to the next color
        color_idx = (color_idx + 1) % len(colors)
    display(HTML(html_output))  # Display HTML output
    print(f"Total number of tokens: {len(tokens)}")
```


```python
highlight_text(API_RESPONSE)

```


key: _node_content
string_value:
{"id_": "6d3ba58f-4b7c-4f4b-8fd4-2a564fd32a77", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55212b9e-0037-4057-8a7c-c74e66111b5d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "14c1ffc369932fb085a52f4bdd94723f41a8dd9acdbec51cae10e2a0bfdeb2eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b2b80524-36ac-4a6a-afa4-ee1d3e1c3dac", "node_type": "1", "metadata": {}, "hash": "a73e9e2da3d3a42a0e17676455ef58d16351d24e1891060c7e7557ecdb02b602", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25639, "end_char_idx": 28690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
PROMPT = """What's the longest word in the English language?"""

API_RESPONSE = get_completion(
    [{"role": "user", "content": PROMPT}], model="gpt-4", logprobs=True, top_logprobs=5
)


def highlight_text(api_response):
    colors = [
        "#FF00FF",  # Magenta
        "#008000",  # Green
        "#FF8C00",  # Dark Orange
        "#FF0000",  # Red
        "#0000FF",  # Blue
    ]
    tokens = api_response.choices[0].logprobs.content

    color_idx = 0  # Initialize color index
    html_output = ""  # Initialize HTML output
    for t in tokens:
        token_str = bytes(t.bytes).decode("utf-8")  # Decode bytes to string

        # Add colored token to HTML output
        html_output += f"<span style='color: {colors[color_idx]}'>{token_str}</span>"

        # Move to the next color
        color_idx = (color_idx + 1) % len(colors)
    display(HTML(html_output))  # Display HTML output
    print(f"Total number of tokens: {len(tokens)}")
```


```python
highlight_text(API_RESPONSE)

```


<span style='color: #FF00FF'>The</span><span style='color: #008000'> longest</span><span style='color: #FF8C00'> word</span><span style='color: #FF0000'> in</span><span style='color: #0000FF'> the</span><span style='color: #FF00FF'> English</span><span style='color: #008000'> language</span><span style='color: #FF8C00'>,</span><span style='color: #FF0000'> according</span><span style='color: #0000FF'> to</span><span style='color: #FF00FF'> the</span><span style='color: #008000'> Guinness</span><span style='color: #FF8C00'> World</span><span style='color: #FF0000'> Records</span><span style='color: #0000FF'>,</span><span style='color: #FF00FF'> is</span><span style='color: #008000'> '</span><span style='color: #FF8C00'>p</span><span style='color: #FF0000'>ne</span><span style='color: #0000FF'>um</span><span style='color: #FF00FF'>on</span><span style='color: #008000'>oul</span><span style='color: #FF8C00'>tram</span><span style='color: #FF0000'>icro</span><span style='color: #0000FF'>sc</span><span style='color: #FF00FF'>op</span><span style='color: #008000'>ics</span><span style='color: #FF8C00'>il</span><span style='color: #FF0000'>ic</span><span style='color: #0000FF'>ov</span><span style='color: #FF00FF'>ol</span><span style='color: #008000'>cano</span><span style='color: #FF8C00'>con</span><span style='color: #FF0000'>iosis</span><span style='color: #0000FF'>'.</span><span style='color: #FF00FF'> It</span><span style='color: #008000'> is</span><span style='color: #FF8C00'> a</span><span style='color: #FF0000'> type</span><span style='color: #0000FF'> of</span><span style='color: #FF00FF'> lung</span><span style='color: #008000'> disease</span><span style='color: #FF8C00'> caused</span><span style='color: #FF0000'> by</span><span style='color: #0000FF'> inh</span><span style='color: #FF00FF'>aling</span><span style='color: #008000'> ash</span><span style='color: #FF8C00'> and</span><span style='color: #FF0000'> sand</span><span style='color: #0000FF'> dust</span><span style='color: #FF00FF'>.</span>


key: _node_content
string_value:
{"id_": "b2b80524-36ac-4a6a-afa4-ee1d3e1c3dac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d3ba58f-4b7c-4f4b-8fd4-2a564fd32a77", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "36d675206e3a044988bf4aa835774e31bb531eb98bd16e5e2c06c26011c2561a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba3cebe3-f234-4a2d-91f2-5d90a8009920", "node_type": "1", "metadata": {}, "hash": "d8916c94644d830c3cca220bf53190ac5f59d86eeb7ae9a84cea7c1fc6232e06", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28697, "end_char_idx": 30663, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Total number of tokens: 51


Next, let's reconstruct a sentence using the bytes parameter. With `logprobs` enabled, we are given both each token and the ASCII (decimal utf-8) values of the token string. These ASCII values can be helpful when handling tokens of or containing emojis or special characters.


```python
PROMPT = """Output the blue heart emoji and its name."""
API_RESPONSE = get_completion(
    [{"role": "user", "content": PROMPT}], model="gpt-4", logprobs=True
)

aggregated_bytes = []
joint_logprob = 0.0

# Iterate over tokens, aggregate bytes and calculate joint logprob
for token in API_RESPONSE.choices[0].logprobs.content:
    print("Token:", token.token)
    print("Log prob:", token.logprob)
    print("Linear prob:", np.round(exp(token.logprob) * 100, 2), "%")
    print("Bytes:", token.bytes, "\n")
    aggregated_bytes += token.bytes
    joint_logprob += token.logprob

# Decode the aggregated bytes to text
aggregated_text = bytes(aggregated_bytes).decode("utf-8")

# Assert that the decoded text is the same as the message content
assert API_RESPONSE.choices[0].message.content == aggregated_text

# Print the results
print("Bytes array:", aggregated_bytes)
print(f"Decoded bytes: {aggregated_text}")
print("Joint prob:", np.round(exp(joint_logprob) * 100, 2), "%")
```

    Token: \xf0\x9f\x92
    Log prob: -0.0003056686
    Linear prob: 99.97 %
    Bytes: [240, 159, 146] 
    
    Token: \x99
    Log prob: 0.0
    Linear prob: 100.0 %
    Bytes: [153] 
    
    Token:  -
    Log prob: -0.0096905725
    Linear prob: 99.04 %
    Bytes: [32, 45] 
    
    Token:  Blue
    Log prob: -0.00042042506
    Linear prob: 99.96 %
    Bytes: [32, 66, 108, 117, 101] 
    
    Token:  Heart
    Log prob: -7.302705e-05
    Linear prob: 99.99 %
    Bytes: [32, 72, 101, 97, 114, 116] 
    
    Bytes array: [240, 159, 146, 153, 32, 45, 32, 66, 108, 117, 101, 32, 72, 101, 97, 114, 116]
    Decoded bytes: 💙 - Blue Heart
    Joint prob: 98.96 %


key: _node_content
string_value:
{"id_": "ba3cebe3-f234-4a2d-91f2-5d90a8009920", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b2b80524-36ac-4a6a-afa4-ee1d3e1c3dac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "00a7ce42477518bf9c6471b1dd1f567046f370a92a5dcb5a0bdaef830a158a12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28543fcd-3255-424e-8066-a45d9bb6a4db", "node_type": "1", "metadata": {}, "hash": "e8807722e9a48b46cc8898a7868543b8bbe73e871a9d60f5a46a9978f9f07271", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30666, "end_char_idx": 32361, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here, we see that while the first token was `\xf0\x9f\x92'`, we can get its ASCII value and append it to a bytes array. Then, we can easily decode this array into a full sentence, and validate with our assert statement that the decoded bytes is the same as our completion message!

Additionally, we can get the joint probability of the entire completion, which is the exponentiated product of each token's log probability. This gives us how `likely` this given completion is given the prompt. Since, our prompt is quite directive (asking for a certain emoji and its name), the joint probability of this output is high! If we ask for a random output however, we'll see a much lower joint probability. This can also be a good tactic for developers during prompt engineering. 

## 5. Calculating perplexity

When looking to assess the model's confidence in a result, it can be useful to calculate perplexity, which is a measure of the uncertainty. Perplexity can be calculated by exponentiating the negative of the average of the logprobs. Generally, a higher perplexity indicates a more uncertain result, and a lower perplexity indicates a more confident result. As such, perplexity can be used to both assess the result of an individual model run and also to compare the relative confidence of results between model runs. While a high confidence doesn't guarantee result accuracy, it can be a helpful signal that can be paired with other evaluation metrics to build a better understanding of your prompt's behavior.

For example, let's say that I want to use `gpt-3.5-turbo` to learn more about artificial intelligence. I could ask a question about recent history and a question about the future:


key: _node_content
string_value:
{"id_": "28543fcd-3255-424e-8066-a45d9bb6a4db", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba3cebe3-f234-4a2d-91f2-5d90a8009920", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "047668dc27cfdaf5169ec8a4bfb7a1b842be984bdbcf7a7c510e47ccaada4b68", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b5b21c7-010f-4492-a679-9bc556e643f0", "node_type": "1", "metadata": {}, "hash": "b67d0455b150122f6a0d07c6728044d9a289dc9d761d82dd202aa1f669cc94ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30666, "end_char_idx": 35517, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here, we see that while the first token was `\xf0\x9f\x92'`, we can get its ASCII value and append it to a bytes array. Then, we can easily decode this array into a full sentence, and validate with our assert statement that the decoded bytes is the same as our completion message!

Additionally, we can get the joint probability of the entire completion, which is the exponentiated product of each token's log probability. This gives us how `likely` this given completion is given the prompt. Since, our prompt is quite directive (asking for a certain emoji and its name), the joint probability of this output is high! If we ask for a random output however, we'll see a much lower joint probability. This can also be a good tactic for developers during prompt engineering. 

## 5. Calculating perplexity

When looking to assess the model's confidence in a result, it can be useful to calculate perplexity, which is a measure of the uncertainty. Perplexity can be calculated by exponentiating the negative of the average of the logprobs. Generally, a higher perplexity indicates a more uncertain result, and a lower perplexity indicates a more confident result. As such, perplexity can be used to both assess the result of an individual model run and also to compare the relative confidence of results between model runs. While a high confidence doesn't guarantee result accuracy, it can be a helpful signal that can be paired with other evaluation metrics to build a better understanding of your prompt's behavior.

For example, let's say that I want to use `gpt-3.5-turbo` to learn more about artificial intelligence. I could ask a question about recent history and a question about the future:


```python
prompts = [
    "In a short sentence, has artifical intelligence grown in the last decade?",
    "In a short sentence, what are your thoughts on the future of artificial intelligence?",
]

for prompt in prompts:
    API_RESPONSE = get_completion(
        [{"role": "user", "content": prompt}],
        model="gpt-3.5-turbo",
        logprobs=True,
    )

    logprobs = [token.logprob for token in API_RESPONSE.choices[0].logprobs.content]
    response_text = API_RESPONSE.choices[0].message.content
    response_text_tokens = [token.token for token in API_RESPONSE.choices[0].logprobs.content]
    max_starter_length = max(len(s) for s in ["Prompt:", "Response:", "Tokens:", "Logprobs:", "Perplexity:"])
    max_token_length = max(len(s) for s in response_text_tokens)
    

    formatted_response_tokens = [s.rjust(max_token_length) for s in response_text_tokens]
    formatted_lps = [f"{lp:.2f}".rjust(max_token_length) for lp in logprobs]

    perplexity_score = np.exp(-np.mean(logprobs))
    print("Prompt:".ljust(max_starter_length), prompt)
    print("Response:".ljust(max_starter_length), response_text, "\n")
    print("Tokens:".ljust(max_starter_length), " ".join(formatted_response_tokens))
    print("Logprobs:".ljust(max_starter_length), " ".join(formatted_lps))
    print("Perplexity:".ljust(max_starter_length), perplexity_score, "\n")
```

    Prompt:     In a short sentence, has artifical intelligence grown in the last decade?
    Response:   Yes, artificial intelligence has grown significantly in the last decade. 
    
    Tokens:                Yes              ,     artificial   intelligence            has          grown  significantly             in            the           last         decade              .
    Logprobs:            -0.00          -0.00          -0.00          -0.00          -0.00          -0.53          -0.11          -0.00          -0.00          -0.01          -0.00          -0.00
    Perplexity: 1.0564125277713383 
    
    Prompt:     In a short sentence, what are your thoughts on the future of artificial intelligence?
    Response:   The future of artificial intelligence holds great potential for transforming industries and improving efficiency, but also raises ethical and societal concerns that must be carefully addressed. 
    
    Tokens:               The        future            of    artificial  intelligence         holds         great     potential           for  transforming    industries           and     improving    efficiency             ,           but          also        raises       ethical           and      societal      concerns          that          must            be     carefully     addressed             .
    Logprobs:           -0.19         -0.03         -0.00         -0.00         -0.00         -0.30         -0.51         -0.24         -0.03         -1.45         -0.23         -0.03         -0.22         -0.83         -0.48         -0.01         -0.38         -0.07         -0.47         -0.63         -0.18         -0.26         -0.01         -0.14         -0.00         -0.59         -0.55         -0.00
    Perplexity: 1.3220795252314004


key: _node_content
string_value:
{"id_": "6b5b21c7-010f-4492-a679-9bc556e643f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f503d917-ed02-45da-83f9-e74ba453d4de", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f314947ff47daa378d1fdb2f27fec83e4b52792457c0fcbdc406108ba67c95fe", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28543fcd-3255-424e-8066-a45d9bb6a4db", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_logprobs.ipynb", "title": "Using logprobs for classification and Q&A evaluation", "tokens": 9243, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e6dcd8ab88126361f4ce86df5f35ae92c2e956ff8dba33304681496b7cedb98d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35526, "end_char_idx": 36498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this example, `gpt-3.5-turbo` returned a lower perplexity score for a more deterministic question about recent history, and a higher perplexity score for a more speculative assessment about the near future. Again, while these differences don't guarantee accuracy, they help point the way for our interpretation of the model's results and our future use of them.

## 6. Conclusion

Nice! We were able to use the `logprobs` parameter to build a more robust classifier, evaluate our retrieval for Q&A system, and encode and decode each 'byte' of our tokens! `logprobs` adds useful information and signal to our completions output, and we are excited to see how developers incorporate it to improve applications.

## 7. Possible extensions

There are many other use cases for `logprobs` that are not covered in this cookbook. We can use `logprobs` for:
  - Moderation
  - Keyword selection
  - Improve prompts and interpretability of outputs
  - Token healing
  - and more!


key: _node_content
string_value:
{"id_": "a953a73d-33cb-44e3-a534-4e3da85036fd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3eda7e1-f2f4-42bd-ae86-39e0b6695fb5", "node_type": "1", "metadata": {}, "hash": "c7eeefa85011d7dd4302d92ec1c7760b942dcad18104a75b93834c92e0d27a97", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Parsing PDF documents for RAG applications

This notebook shows how to leverage GPT-4V to turn rich PDF documents such as slide decks or exports from web pages into usable content for your RAG application.

This technique can be used if you have a lot of unstructured data containing valuable information that you want to be able to retrieve as part of your RAG pipeline.

For example, you could build a Knowledge Assistant that could answer user queries about your company or product based on information contained in PDF documents. 

The example documents used in this notebook are located at [data/example_pdfs](data/example_pdfs). They are related to OpenAI's APIs and various techniques that can be used as part of LLM projects.

## Data preparation

In this section, we will process our input data to prepare it for retrieval.

We will do this in 2 ways:

1. Extracting text with pdfminer
2. Converting the PDF pages to images to analyze them with GPT-4V

You can skip the 1st method if you want to only use the content inferred from the image analysis.

### Setup

We need to install a few libraries to convert the PDF to images and extract the text (optional).

**Note: You need to install `poppler` on your machine for the `pdf2image` library to work. You can follow the instructions to install it [here](https://pypi.org/project/pdf2image/).**


```python
%pip install pdf2image
%pip install pdfminer
%pip install openai
%pip install scikit-learn
%pip install rich
%pip install tqdm
%pip install concurrent
```


```python
# Imports
from pdf2image import convert_from_path
from pdf2image.exceptions import (
    PDFInfoNotInstalledError,
    PDFPageCountError,
    PDFSyntaxError
)
from pdfminer.high_level import extract_text
import base64
from io import BytesIO
import os
import concurrent
from tqdm import tqdm
from openai import OpenAI
import re
import pandas as pd 
from sklearn.metrics.pairwise import cosine_similarity
import json
import numpy as np
from rich import print
from ast import literal_eval
```

### File processing


```python
def convert_doc_to_images(path):
    images = convert_from_path(path)
    return images

def extract_text_from_doc(path):
    text = extract_text(path)
    page_text = []
    return text
```

#### Testing with an example


```python
file_path = "data/example_pdfs/fine-tuning-deck.pdf"

images = convert_doc_to_images(file_path)
```


```python
text = extract_text_from_doc(file_path)
```


```python
for img in images:
    display(img)
```


    
![png](output_10_0.png)
    



    
![png](output_10_1.png)
    



    
![png](output_10_2.png)
    



    
![png](output_10_3.png)
    



    
![png](output_10_4.png)
    



    
![png](output_10_5.png)
    



    
![png](output_10_6.png)
    


### Image analysis with GPT-4V

After converting a PDF file to multiple images, we'll use GPT-4V to analyze the content based on the images.


```python
# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python
client = OpenAI()
```


key: _node_content
string_value:
{"id_": "d3eda7e1-f2f4-42bd-ae86-39e0b6695fb5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a953a73d-33cb-44e3-a534-4e3da85036fd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "473e7fcae51fbd80c627de7d73e994cc85a692a7cedf5f67bf51983eec7427c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96c58251-d9f8-4b0c-a627-43ab31208209", "node_type": "1", "metadata": {}, "hash": "5cfb408248c7d959a7df1682ee28cf59fa622df94eb22bbd1300654c288a9cc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1525, "end_char_idx": 3373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Imports
from pdf2image import convert_from_path
from pdf2image.exceptions import (
    PDFInfoNotInstalledError,
    PDFPageCountError,
    PDFSyntaxError
)
from pdfminer.high_level import extract_text
import base64
from io import BytesIO
import os
import concurrent
from tqdm import tqdm
from openai import OpenAI
import re
import pandas as pd 
from sklearn.metrics.pairwise import cosine_similarity
import json
import numpy as np
from rich import print
from ast import literal_eval
```

### File processing


```python
def convert_doc_to_images(path):
    images = convert_from_path(path)
    return images

def extract_text_from_doc(path):
    text = extract_text(path)
    page_text = []
    return text
```

#### Testing with an example


```python
file_path = "data/example_pdfs/fine-tuning-deck.pdf"

images = convert_doc_to_images(file_path)
```


```python
text = extract_text_from_doc(file_path)
```


```python
for img in images:
    display(img)
```


    
![png](output_10_0.png)
    



    
![png](output_10_1.png)
    



    
![png](output_10_2.png)
    



    
![png](output_10_3.png)
    



    
![png](output_10_4.png)
    



    
![png](output_10_5.png)
    



    
![png](output_10_6.png)
    


### Image analysis with GPT-4V

After converting a PDF file to multiple images, we'll use GPT-4V to analyze the content based on the images.


```python
# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python
client = OpenAI()
```


```python
# Converting images to base64 encoded images in a data URI format to use with the ChatCompletions API
def get_img_uri(img):
    buffer = BytesIO()
    img.save(buffer, format="jpeg")
    base64_image = base64.b64encode(buffer.getvalue()).decode("utf-8")
    data_uri = f"data:image/jpeg;base64,{base64_image}"
    return data_uri
```


key: _node_content
string_value:
{"id_": "96c58251-d9f8-4b0c-a627-43ab31208209", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3eda7e1-f2f4-42bd-ae86-39e0b6695fb5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c71962f6b4b7bb6d025b8dcedd5a3ddd40708edb33400ad75bc058539728db1b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f37a7868-bbd9-4426-99c4-c0d42152a24d", "node_type": "1", "metadata": {}, "hash": "6e6b018820e2f2be80becda6bbd93273637f2392d6ae23d9e464d881f68079dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2048, "end_char_idx": 5377, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def convert_doc_to_images(path):
    images = convert_from_path(path)
    return images

def extract_text_from_doc(path):
    text = extract_text(path)
    page_text = []
    return text
```

#### Testing with an example


```python
file_path = "data/example_pdfs/fine-tuning-deck.pdf"

images = convert_doc_to_images(file_path)
```


```python
text = extract_text_from_doc(file_path)
```


```python
for img in images:
    display(img)
```


    
![png](output_10_0.png)
    



    
![png](output_10_1.png)
    



    
![png](output_10_2.png)
    



    
![png](output_10_3.png)
    



    
![png](output_10_4.png)
    



    
![png](output_10_5.png)
    



    
![png](output_10_6.png)
    


### Image analysis with GPT-4V

After converting a PDF file to multiple images, we'll use GPT-4V to analyze the content based on the images.


```python
# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python
client = OpenAI()
```


```python
# Converting images to base64 encoded images in a data URI format to use with the ChatCompletions API
def get_img_uri(img):
    buffer = BytesIO()
    img.save(buffer, format="jpeg")
    base64_image = base64.b64encode(buffer.getvalue()).decode("utf-8")
    data_uri = f"data:image/jpeg;base64,{base64_image}"
    return data_uri
```


```python
system_prompt = '''
You will be provided with an image of a pdf page or a slide. Your goal is to talk about the content that you see, in technical terms, as if you were delivering a presentation.

If there are diagrams, describe the diagrams and explain their meaning.
For example: if there is a diagram describing a process flow, say something like "the process flow starts with X then we have Y and Z..."

If there are tables, describe logically the content in the tables
For example: if there is a table listing items and prices, say something like "the prices are the following: A for X, B for Y..."

DO NOT include terms referring to the content format
DO NOT mention the content type - DO focus on the content itself
For example: if there is a diagram/chart and text on the image, talk about both without mentioning that one is a chart and the other is text.
Simply describe what you see in the diagram and what you understand from the text.

You should keep it concise, but keep in mind your audience cannot see the image so be exhaustive in describing the content.

Exclude elements that are not relevant to the content:
DO NOT mention page numbers or the position of the elements on the image.

------

If there is an identifiable title, identify the title to give the output in the following format:

{TITLE}

{Content description}

If there is no clear title, simply return the content description.

'''

def analyze_image(img_url):
    response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    temperature=0,
    messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": img_url,
                },
            ],
        }
    ],
        max_tokens=300,
        top_p=0.1
    )

    return response.choices[0].message.content
```

#### Testing with an example


key: _node_content
string_value:
{"id_": "f37a7868-bbd9-4426-99c4-c0d42152a24d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96c58251-d9f8-4b0c-a627-43ab31208209", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8b4ce093739f8218730b518e57e198fd2855a7a1c9f367eafb21dc918c44ff27", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "08e2bdb1-792c-42f2-ad36-f01605b1d952", "node_type": "1", "metadata": {}, "hash": "6380550b509f50b7025d0eefcd638a3a1644c3d0eb7dc1393df8bab16d380beb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5380, "end_char_idx": 8217, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
img = images[2]
data_uri = get_img_uri(img)
```


```python
res = analyze_image(data_uri)
print(res)
```

    What is Fine-tuning
    
    Fine-tuning a model consists of training the model to follow a set of given input/output examples. This will teach the model to behave in a certain way when confronted with a similar input in the future.
    
    We recommend using 50-100 examples even if the minimum is 10.
    
    The process involves starting with a public model, using training data to train the model, and resulting in a fine-tuned model.


#### Processing all documents


```python
files_path = "data/example_pdfs"

all_items = os.listdir(files_path)
files = [item for item in all_items if os.path.isfile(os.path.join(files_path, item))]
```


```python
def analyze_doc_image(img):
    img_uri = get_img_uri(img)
    data = analyze_image(img_uri)
    return data
```

We will list all files in the example folder and process them by 
1. Extracting the text
2. Converting the docs to images
3. Analyzing pages with GPT-4V

Note: This takes about ~2 mins to run. Feel free to skip and load directly the result file (see below).


```python
docs = []

for f in files:
    
    path = f"{files_path}/{f}"
    doc = {
        "filename": f
    }
    text = extract_text_from_doc(path)
    doc['text'] = text
    imgs = convert_doc_to_images(path)
    pages_description = []
    
    print(f"Analyzing pages for doc {f}")
    
    # Concurrent execution
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        
        # Removing 1st slide as it's usually just an intro
        futures = [
            executor.submit(analyze_doc_image, img)
            for img in imgs[1:]
        ]
        
        with tqdm(total=len(imgs)-1) as pbar:
            for _ in concurrent.futures.as_completed(futures):
                pbar.update(1)
        
        for f in futures:
            res = f.result()
            pages_description.append(res)
        
    doc['pages_description'] = pages_description
    docs.append(doc)
```

    Analyzing pages for doc rag-deck.pdf


    100%|██████████████████████████████████████████████████████████████████| 19/19 [00:32<00:00,  1.72s/it]


    Analyzing pages for doc models-page.pdf


    100%|████████████████████████████████████████████████████████████████████| 9/9 [00:25<00:00,  2.80s/it]


    Analyzing pages for doc evals-decks.pdf


    100%|██████████████████████████████████████████████████████████████████| 12/12 [00:29<00:00,  2.44s/it]


    Analyzing pages for doc fine-tuning-deck.pdf


    100%|████████████████████████████████████████████████████████████████████| 6/6 [00:19<00:00,  3.32s/it]



```python
# Saving result to file for later
json_path = "data/parsed_pdf_docs.json"

with open(json_path, 'w') as f:
    json.dump(docs, f)
```


key: _node_content
string_value:
{"id_": "08e2bdb1-792c-42f2-ad36-f01605b1d952", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f37a7868-bbd9-4426-99c4-c0d42152a24d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "62225f2ed2783f9575b1fe0eeef336ef89b3401bc7d3e4e5e9437e2a0e03926a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "736dfb0a-3805-4e35-a491-71b316a22643", "node_type": "1", "metadata": {}, "hash": "1448e142e45b970a5e8d1c4f77bb7f13d5dafa5f036151a89a93f56e412e0bd6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7494, "end_char_idx": 10469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
100%|██████████████████████████████████████████████████████████████████| 19/19 [00:32<00:00,  1.72s/it]


    Analyzing pages for doc models-page.pdf


    100%|████████████████████████████████████████████████████████████████████| 9/9 [00:25<00:00,  2.80s/it]


    Analyzing pages for doc evals-decks.pdf


    100%|██████████████████████████████████████████████████████████████████| 12/12 [00:29<00:00,  2.44s/it]


    Analyzing pages for doc fine-tuning-deck.pdf


    100%|████████████████████████████████████████████████████████████████████| 6/6 [00:19<00:00,  3.32s/it]



```python
# Saving result to file for later
json_path = "data/parsed_pdf_docs.json"

with open(json_path, 'w') as f:
    json.dump(docs, f)
```


```python
# Optional: load content from the saved file
with open(json_path, 'r') as f:
    docs = json.load(f)
```

### Embedding content
Before embedding the content, we will chunk it logically by page.
For real-world scenarios, you could explore more advanced ways to chunk the content:
- Cutting it into smaller pieces
- Adding data - such as the slide title, deck title and/or the doc description - at the beginning of each piece of content. That way, each independent chunk can be in context

For the sake of brevity, we will use a very simple chunking strategy and rely on separators to split the text by page.


```python
# Chunking content by page and merging together slides text & description if applicable
content = []
for doc in docs:
    # Removing first slide as well
    text = doc['text'].split('\f')[1:]
    description = doc['pages_description']
    description_indexes = []
    for i in range(len(text)):
        slide_content = text[i] + '\n'
        # Trying to find matching slide description
        slide_title = text[i].split('\n')[0]
        for j in range(len(description)):
            description_title = description[j].split('\n')[0]
            if slide_title.lower() == description_title.lower():
                slide_content += description[j].replace(description_title, '')
                # Keeping track of the descriptions added
                description_indexes.append(j)
        # Adding the slide content + matching slide description to the content pieces
        content.append(slide_content) 
    # Adding the slides descriptions that weren't used
    for j in range(len(description)):
        if j not in description_indexes:
            content.append(description[j])
```


```python
for c in content:
    print(c)
    print("\n\n-------------------------------\n\n")
```


```python
# Cleaning up content
# Removing trailing spaces, additional line breaks, page numbers and references to the content being a slide
clean_content = []
for c in content:
    text = c.replace(' \n', '').replace('\n\n', '\n').replace('\n\n\n', '\n').strip()
    text = re.sub(r"(?<=\n)\d{1,2}", "", text)
    text = re.sub(r"\b(?:the|this)\s*slide\s*\w+\b", "", text, flags=re.IGNORECASE)
    clean_content.append(text)
```


key: _node_content
string_value:
{"id_": "736dfb0a-3805-4e35-a491-71b316a22643", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "08e2bdb1-792c-42f2-ad36-f01605b1d952", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9fdef58cfae9e23526cef920574e33670a0ddd006628c5dd078175f4fc4699f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8e1c055-d359-4d4e-941e-8e4eca4d6054", "node_type": "1", "metadata": {}, "hash": "5d213883a60134bfc6550b9a373135697b822225044e763240c4c6300b1f466b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8839, "end_char_idx": 11748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Chunking content by page and merging together slides text & description if applicable
content = []
for doc in docs:
    # Removing first slide as well
    text = doc['text'].split('\f')[1:]
    description = doc['pages_description']
    description_indexes = []
    for i in range(len(text)):
        slide_content = text[i] + '\n'
        # Trying to find matching slide description
        slide_title = text[i].split('\n')[0]
        for j in range(len(description)):
            description_title = description[j].split('\n')[0]
            if slide_title.lower() == description_title.lower():
                slide_content += description[j].replace(description_title, '')
                # Keeping track of the descriptions added
                description_indexes.append(j)
        # Adding the slide content + matching slide description to the content pieces
        content.append(slide_content) 
    # Adding the slides descriptions that weren't used
    for j in range(len(description)):
        if j not in description_indexes:
            content.append(description[j])
```


```python
for c in content:
    print(c)
    print("\n\n-------------------------------\n\n")
```


```python
# Cleaning up content
# Removing trailing spaces, additional line breaks, page numbers and references to the content being a slide
clean_content = []
for c in content:
    text = c.replace(' \n', '').replace('\n\n', '\n').replace('\n\n\n', '\n').strip()
    text = re.sub(r"(?<=\n)\d{1,2}", "", text)
    text = re.sub(r"\b(?:the|this)\s*slide\s*\w+\b", "", text, flags=re.IGNORECASE)
    clean_content.append(text)
```


```python
for c in clean_content:
    print(c)
    print("\n\n-------------------------------\n\n")
```


```python
# Creating the embeddings
# We'll save to a csv file here for testing purposes but this is where you should load content in your vectorDB.
df = pd.DataFrame(clean_content, columns=['content'])
print(df.shape)
df.head()
```

    (64, 1)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Overview\nRetrieval-Augmented Generationenhanc...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What is RAG\nRetrieve information to Augment t...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>When to use RAG\nGood for  ✅\nNot good for  ❌\...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Technical patterns\nData preparation\nInput pr...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Technical patterns\nData preparation\nchunk do...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "e8e1c055-d359-4d4e-941e-8e4eca4d6054", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "736dfb0a-3805-4e35-a491-71b316a22643", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7d445c9bdc2e4645a558e76f6f7e4fff1c1b0e17a44c0301fc4fbc004c3f7259", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "516d4777-57fe-453a-8814-d36fd4cec5a2", "node_type": "1", "metadata": {}, "hash": "3186014fa5e279eac0d2e31db300695c60b087423678dec13e4c0e752fc067a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10472, "end_char_idx": 12106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
for c in clean_content:
    print(c)
    print("\n\n-------------------------------\n\n")
```


```python
# Creating the embeddings
# We'll save to a csv file here for testing purposes but this is where you should load content in your vectorDB.
df = pd.DataFrame(clean_content, columns=['content'])
print(df.shape)
df.head()
```

    (64, 1)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Overview\nRetrieval-Augmented Generationenhanc...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What is RAG\nRetrieve information to Augment t...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>When to use RAG\nGood for  ✅\nNot good for  ❌\...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Technical patterns\nData preparation\nInput pr...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Technical patterns\nData preparation\nchunk do...</td>
    </tr>
  </tbody>
</table>
</div>




```python
embeddings_model = "text-embedding-3-large"

def get_embeddings(text):
    embeddings = client.embeddings.create(
      model="text-embedding-3-small",
      input=text,
      encoding_format="float"
    )
    return embeddings.data[0].embedding
```


```python
df['embeddings'] = df['content'].apply(lambda x: get_embeddings(x))
df.head()
```


key: _node_content
string_value:
{"id_": "516d4777-57fe-453a-8814-d36fd4cec5a2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8e1c055-d359-4d4e-941e-8e4eca4d6054", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d0a833dc928129834c6b7a1fffd6675b7a22e0f36ca03695f284e19933c1cc69", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d332d9c-a890-455b-86fd-9f20af915d15", "node_type": "1", "metadata": {}, "hash": "c79f64858aa4447d9f6d3e2547146f3f3af408a469a872b10271ce5767cce71b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10829, "end_char_idx": 13381, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Overview\nRetrieval-Augmented Generationenhanc...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What is RAG\nRetrieve information to Augment t...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>When to use RAG\nGood for  ✅\nNot good for  ❌\...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Technical patterns\nData preparation\nInput pr...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Technical patterns\nData preparation\nchunk do...</td>
    </tr>
  </tbody>
</table>
</div>




```python
embeddings_model = "text-embedding-3-large"

def get_embeddings(text):
    embeddings = client.embeddings.create(
      model="text-embedding-3-small",
      input=text,
      encoding_format="float"
    )
    return embeddings.data[0].embedding
```


```python
df['embeddings'] = df['content'].apply(lambda x: get_embeddings(x))
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>content</th>
      <th>embeddings</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Overview\nRetrieval-Augmented Generationenhanc...</td>
      <td>[-0.014744381, 0.03017278, 0.06353764, 0.02110...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>What is RAG\nRetrieve information to Augment t...</td>
      <td>[-0.024337867, 0.022921458, -0.00971687, 0.010...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>When to use RAG\nGood for  ✅\nNot good for  ❌\...</td>
      <td>[-0.011084231, 0.021158217, -0.00430421, 0.017...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Technical patterns\nData preparation\nInput pr...</td>
      <td>[-0.0058343858, 0.0408407, 0.054318383, 0.0190...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Technical patterns\nData preparation\nchunk do...</td>
      <td>[-0.010359385, 0.03736894, 0.052995477, 0.0180...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "9d332d9c-a890-455b-86fd-9f20af915d15", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "516d4777-57fe-453a-8814-d36fd4cec5a2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d5d3e15ae1f0f52a487704a3e692842c44155de88066c947fe7629a7dd393220", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "137ccf0b-17ef-4cc8-9cc5-d35e854f393c", "node_type": "1", "metadata": {}, "hash": "070101153f482981e2cf37960ce5d0056e0a0a1fb78c08ef2408d4c653157cee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13386, "end_char_idx": 16614, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Saving locally for later
data_path = "data/parsed_pdf_docs_with_embeddings.csv"
df.to_csv(data_path, index=False)
```


```python
# Optional: load data from saved file
df = pd.read_csv(data_path)
df["embeddings"] = df.embeddings.apply(literal_eval).apply(np.array)
```

## Retrieval-augmented generation

The last step of the process is to generate outputs in response to input queries, after retrieving content as context to reply.


```python
system_prompt = '''
    You will be provided with an input prompt and content as context that can be used to reply to the prompt.
    
    You will do 2 things:
    
    1. First, you will internally assess whether the content provided is relevant to reply to the input prompt. 
    
    2a. If that is the case, answer directly using this content. If the content is relevant, use elements found in the content to craft a reply to the input prompt.

    2b. If the content is not relevant, use your own knowledge to reply or say that you don't know how to respond if your knowledge is not sufficient to answer.
    
    Stay concise with your answer, replying specifically to the input prompt without mentioning additional information provided in the context content.
'''

model="gpt-4-turbo-preview"

def search_content(df, input_text, top_k):
    embedded_value = get_embeddings(input_text)
    df["similarity"] = df.embeddings.apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    res = df.sort_values('similarity', ascending=False).head(top_k)
    return res

def get_similarity(row):
    similarity_score = row['similarity']
    if isinstance(similarity_score, np.ndarray):
        similarity_score = similarity_score[0][0]
    return similarity_score

def generate_output(input_prompt, similar_content, threshold = 0.5):
    
    content = similar_content.iloc[0]['content']
    
    # Adding more matching content if the similarity is above threshold
    if len(similar_content) > 1:
        for i, row in similar_content.iterrows():
            similarity_score = get_similarity(row)
            if similarity_score > threshold:
                content += f"\n\n{row['content']}"
            
    prompt = f"INPUT PROMPT:\n{input_prompt}\n-------\nCONTENT:\n{content}"
    
    completion = client.chat.completions.create(
        model=model,
        temperature=0.5,
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": prompt
            }
        ]
    )

    return completion.choices[0].message.content
```


```python
# Example user queries related to the content
example_inputs = [
    'What are the main models you offer?',
    'Do you have a speech recognition model?',
    'Which embedding model should I use for non-English use cases?',
    'Can I introduce new knowledge in my LLM app using RAG?',
    'How many examples do I need to fine-tune a model?',
    'Which metric can I use to evaluate a summarization task?',
    'Give me a detailed example for an evaluation process where we are looking for a clear answer to compare to a ground truth.',
]
```


key: _node_content
string_value:
{"id_": "137ccf0b-17ef-4cc8-9cc5-d35e854f393c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d332d9c-a890-455b-86fd-9f20af915d15", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1250669e2cf1d4badf1a394ee42bf6ff2f0e9207672991895a1e99b6b0a57488", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37bb8443-2d69-4a95-95f0-041dbff2c22f", "node_type": "1", "metadata": {}, "hash": "803323320ba445ff42ded6ac759b8016c07198eac81676108b44bb86bc8b0943", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16062, "end_char_idx": 18235, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Example user queries related to the content
example_inputs = [
    'What are the main models you offer?',
    'Do you have a speech recognition model?',
    'Which embedding model should I use for non-English use cases?',
    'Can I introduce new knowledge in my LLM app using RAG?',
    'How many examples do I need to fine-tune a model?',
    'Which metric can I use to evaluate a summarization task?',
    'Give me a detailed example for an evaluation process where we are looking for a clear answer to compare to a ground truth.',
]
```


```python
# Running the RAG pipeline on each example
for ex in example_inputs:
    print(f"[deep_pink4][bold]QUERY:[/bold] {ex}[/deep_pink4]\n\n")
    matching_content = search_content(df, ex, 3)
    print(f"[grey37][b]Matching content:[/b][/grey37]\n")
    for i, match in matching_content.iterrows():
        print(f"[grey37][i]Similarity: {get_similarity(match):.2f}[/i][/grey37]")
        print(f"[grey37]{match['content'][:100]}{'...' if len(match['content']) > 100 else ''}[/[grey37]]\n\n")
    reply = generate_output(ex, matching_content)
    print(f"[turquoise4][b]REPLY:[/b][/turquoise4]\n\n[spring_green4]{reply}[/spring_green4]\n\n--------------\n\n")
```


<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> What are the main models you offer?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.43</span>
</pre>


key: _node_content
string_value:
{"id_": "37bb8443-2d69-4a95-95f0-041dbff2c22f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "137ccf0b-17ef-4cc8-9cc5-d35e854f393c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "82fb908d2c5006acf6a3b9057a0a3205cf03c142394d86a1c016a63b4cdd5d60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "edef560f-dee2-44bc-bff7-82d6321b90c5", "node_type": "1", "metadata": {}, "hash": "d9048fdececaa50e7bf31eb1efab831215f7983da756c843045a7f33112205b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17287, "end_char_idx": 19292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> What are the main models you offer?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.43</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content lists various API endpoints and their corresponding latest models:</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">-...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.39</span>
</pre>


key: _node_content
string_value:
{"id_": "edef560f-dee2-44bc-bff7-82d6321b90c5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37bb8443-2d69-4a95-95f0-041dbff2c22f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c86dd316fe08ca30281c5abe4d13408454029954fc3e2b674d4e36ececc31332", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "597f7517-6c2e-43e9-b8ee-490d6d2498a3", "node_type": "1", "metadata": {}, "hash": "a8d416c1cafe807a705cd70c37f48121dadf80f225fe176f8da6d68cfa99b4a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18240, "end_char_idx": 20467, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content lists various API endpoints and their corresponding latest models:</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">-...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.39</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">26</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">02</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">2024</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">, </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">17:58</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The Moderation models are designed to check whether content co...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


key: _node_content
string_value:
{"id_": "597f7517-6c2e-43e9-b8ee-490d6d2498a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "edef560f-dee2-44bc-bff7-82d6321b90c5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "729b27389e17cb3f1a10dc7c6c4e729078d310d8815b512a3582f12911f9110e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "006c53de-ff2f-4a85-a9e2-e24ddcd85a50", "node_type": "1", "metadata": {}, "hash": "ac905c9f39fad5867e873d1d1eb6a5f494a4c4c98f3eaa525cc3954e01529303", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20470, "end_char_idx": 22866, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.39</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models provided by OpenAI, focusing on moderation models and GPT base ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">The main models we offer include:</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For completions: gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo-instruct, babbage-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">, and davinci-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For embeddings: text-embedding-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3</span><span style="color: #00875f; text-decoration-color: #00875f">-small, text-embedding-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3</span><span style="color: #00875f; text-decoration-color: #00875f">-large, and text-embedding-ada-</span><span style="color: #00875f; text-decoration-color: #00875f;


key: _node_content
string_value:
{"id_": "006c53de-ff2f-4a85-a9e2-e24ddcd85a50", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "597f7517-6c2e-43e9-b8ee-490d6d2498a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f87c9a2882b534d18908dae832f36033f009f9528bf58ce0ab042324f207af36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e6e9a94-2699-470b-8953-9c23b71ec80e", "node_type": "1", "metadata": {}, "hash": "7a094f28bf2fcaa6a66a30c57cb265a28a20b4ac815915c71ac8b1f4a317b489", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21597, "end_char_idx": 24044, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">The main models we offer include:</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For completions: gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo-instruct, babbage-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">, and davinci-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For embeddings: text-embedding-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3</span><span style="color: #00875f; text-decoration-color: #00875f">-small, text-embedding-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3</span><span style="color: #00875f; text-decoration-color: #00875f">-large, and text-embedding-ada-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For fine-tuning jobs: gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo, babbage-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">, and davinci-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For moderations: text-moderation-stable and text-moderation.</span>
<span style="color: #00875f; text-decoration-color: #00875f">Additionally, we have the latest models like gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo-16k and fine-tuned versions of gpt-</span><span style="color: #00875f;


key: _node_content
string_value:
{"id_": "8e6e9a94-2699-470b-8953-9c23b71ec80e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "006c53de-ff2f-4a85-a9e2-e24ddcd85a50", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6584b30a0ebf41d73084fa67f7108ec5528baa16d14d3bd394b58e922a12ec20", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "272920f6-3106-44f3-85ac-f6ccc6ecc64a", "node_type": "1", "metadata": {}, "hash": "e5be8d308086f9a8a5de4522146e583bf1481aeaa0abd179ddca9daaef9a213c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22835, "end_char_idx": 25163, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For fine-tuning jobs: gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo, babbage-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">, and davinci-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">002</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>
<span style="color: #00875f; text-decoration-color: #00875f">- For moderations: text-moderation-stable and text-moderation.</span>
<span style="color: #00875f; text-decoration-color: #00875f">Additionally, we have the latest models like gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo-16k and fine-tuned versions of gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Do you have a speech recognition model?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.53</span>
</pre>


key: _node_content
string_value:
{"id_": "272920f6-3106-44f3-85ac-f6ccc6ecc64a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e6e9a94-2699-470b-8953-9c23b71ec80e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "aa6fe654abee4c46a5e3156f547016ff93ba6498a20adb8559555a0a828834cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58b5e339-12a1-4cd2-9106-69f8b35f27da", "node_type": "1", "metadata": {}, "hash": "de9f03a23d21939c8ee51742c77a36155acdc90fa6b6d8975f05fb7400d31c9c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23877, "end_char_idx": 26084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo-16k and fine-tuned versions of gpt-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3.5</span><span style="color: #00875f; text-decoration-color: #00875f">-turbo.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Do you have a speech recognition model?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.53</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models related to text-to-speech, speech recognition, embeddings, and ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>


key: _node_content
string_value:
{"id_": "58b5e339-12a1-4cd2-9106-69f8b35f27da", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "272920f6-3106-44f3-85ac-f6ccc6ecc64a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "76b1dddf385429735ae52c05795b0dde1040a9646a2bf5f73b2358153f0eef89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "917d79e5-2aa8-4419-9678-ebf714066cad", "node_type": "1", "metadata": {}, "hash": "a15ed8af85624a429687178daef5c2c1a08c97797cee34d9556a4c8c6941ee1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25168, "end_char_idx": 27705, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models related to text-to-speech, speech recognition, embeddings, and ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">26</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">02</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">2024</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">, </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">17:58</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">MODEL</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">DE S CRIPTION</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">tts-</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">1</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">New  Text-to-speech </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">1</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The latest tex...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


key: _node_content
string_value:
{"id_": "917d79e5-2aa8-4419-9678-ebf714066cad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58b5e339-12a1-4cd2-9106-69f8b35f27da", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7c7302dfb6b74884a9e965a1bd6c9d85153191ee7a2420566e12edc860c9f91b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91e35b5e-13dc-4665-aab5-ff9a6b347008", "node_type": "1", "metadata": {}, "hash": "19fd98d37c86cb6136cacb44fe66023c9cf3b2a3ea0b9aea05098cae62526de6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27708, "end_char_idx": 29595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.44</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">26</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">02</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">2024</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">, </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">17:58</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">ENDP OINT</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">DATA USED</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">FOR TRAINING</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">DEFAULT</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">RETENTION</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">ELIGIBLE FO...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "91e35b5e-13dc-4665-aab5-ff9a6b347008", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "917d79e5-2aa8-4419-9678-ebf714066cad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b5d7d1962e406da35faf050bf5fdf35afe7e1c19ac3b97d3cc9556560ac39bd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26d55122-c1d5-401d-9330-e31f9736c5c7", "node_type": "1", "metadata": {}, "hash": "d86cdf3a11a6bb310918e5aab51b22ba7ca96924281cb1c3f4cdf92b859f1cc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29589, "end_char_idx": 32043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">Yes, the Whisper model is a general-purpose speech recognition model mentioned in the content, capable of </span>
<span style="color: #00875f; text-decoration-color: #00875f">multilingual speech recognition, speech translation, and language identification. The v2-large model, referred to </span>
<span style="color: #00875f; text-decoration-color: #00875f">as </span><span style="color: #00875f; text-decoration-color: #00875f">"whisper-1"</span><span style="color: #00875f; text-decoration-color: #00875f">, is available through an API and is optimized for faster performance.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Which embedding model should I use for non-English use cases?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.57</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models related to text-to-speech, speech recognition, embeddings, and ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "26d55122-c1d5-401d-9330-e31f9736c5c7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "91e35b5e-13dc-4665-aab5-ff9a6b347008", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c6421d35d80ef859b93278953e6c75ae5869200db0b2746e1735fc5d08d3ef9b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3d8a0b5e-4142-434c-895e-4c6d5d003d1c", "node_type": "1", "metadata": {}, "hash": "b5aff794b9108b65dde78bd3ca80ceda270860ff317ccbcde9269b5054f788bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30865, "end_char_idx": 32400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.57</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models related to text-to-speech, speech recognition, embeddings, and ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.46</span>
</pre>


key: _node_content
string_value:
{"id_": "3d8a0b5e-4142-434c-895e-4c6d5d003d1c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26d55122-c1d5-401d-9330-e31f9736c5c7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3631a930bfc2aa925e450fda98ba4b6bfda505250015b282fdb688fe686c7a58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba8b8594-f92a-4a72-ae7f-deeb1b54b302", "node_type": "1", "metadata": {}, "hash": "cad3e2b0b03ca2adbb162077da1a846065b401d6069eaae2694bc1c9b666ef12", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31484, "end_char_idx": 34021, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The content describes various models related to text-to-speech, speech recognition, embeddings, and ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.46</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">26</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">02</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">2024</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">, </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">17:58</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">MODEL</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">DE S CRIPTION</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">tts-</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">1</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">New  Text-to-speech </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">1</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">The latest tex...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


key: _node_content
string_value:
{"id_": "ba8b8594-f92a-4a72-ae7f-deeb1b54b302", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3d8a0b5e-4142-434c-895e-4c6d5d003d1c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f39e31f519d34260c13b39a5cbafdb97520b5e0bf23b2473c3a3cee9d0a7a8c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e72a43e2-9c15-408e-ae01-e0c3348c2a95", "node_type": "1", "metadata": {}, "hash": "0aac3eae59addae4e703accba947b48b023bc827ac42c7e85ffe75a09d41d92c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34024, "end_char_idx": 35794, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.40</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">26</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">02</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">2024</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">, </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">17:58</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Models - OpenAI API</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Multilingual capabilities</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">GPT-</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">4</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f"> outperforms both previous larg...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "e72a43e2-9c15-408e-ae01-e0c3348c2a95", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba8b8594-f92a-4a72-ae7f-deeb1b54b302", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c9dfc1c53e7bdcfaca38d910367db028d756101d3c81341094727ce8f1c03c95", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cb7cedfc-d828-4e3b-9994-101810012921", "node_type": "1", "metadata": {}, "hash": "aa9077f13241d95863161de656c028f4d1a2e0aebe3671457fac577b59bde19e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35788, "end_char_idx": 37795, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">For non-English use cases, you should use the </span><span style="color: #00875f; text-decoration-color: #00875f">"V3 large"</span><span style="color: #00875f; text-decoration-color: #00875f"> embedding model, as it is described as the most capable </span>
<span style="color: #00875f; text-decoration-color: #00875f">for both English and non-English tasks, with an output dimension of </span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">3</span><span style="color: #00875f; text-decoration-color: #00875f">,</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">072</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Can I introduce new knowledge in my LLM app using RAG?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>


key: _node_content
string_value:
{"id_": "cb7cedfc-d828-4e3b-9994-101810012921", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e72a43e2-9c15-408e-ae01-e0c3348c2a95", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "99dc4096f0b877b24ed065a0b4c57195e98af28ea68ee8145a968e8932740da5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e36cf28d-f852-41d1-8eb6-93c6db8e6b2d", "node_type": "1", "metadata": {}, "hash": "9b2f55dd8c2c238cf83b7a7520048ec39b972a8e3633846abd80328cfa5196e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36817, "end_char_idx": 38852, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Can I introduce new knowledge in my LLM app using RAG?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What is RAG</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Retrieve information to Augment the model’s knowledge and Generate the output</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">“What is y...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.49</span>
</pre>


key: _node_content
string_value:
{"id_": "e36cf28d-f852-41d1-8eb6-93c6db8e6b2d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb7cedfc-d828-4e3b-9994-101810012921", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b3245080a02b4c3b4d33a0c073030cf2838a33cea5b5fe3c68b987e21fa1272a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e085b01-14c8-4cd5-9ab7-8d71c72e6b75", "node_type": "1", "metadata": {}, "hash": "f9eb8763ba491b4ae4c80a7b6363ff234a6dffa7a5f2de3d1857b5f5814db18a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37800, "end_char_idx": 39892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What is RAG</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Retrieve information to Augment the model’s knowledge and Generate the output</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">“What is y...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.49</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">When to use RAG</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Good for  ✅</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Not good for  ❌</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Introducing new information to the model</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Teaching ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "3e085b01-14c8-4cd5-9ab7-8d71c72e6b75", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e36cf28d-f852-41d1-8eb6-93c6db8e6b2d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "5335b46fbc5e212c1ec10a5eb52d664fa3903da2889a0fe716254988c0feec41", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e641843-1d7f-417e-a3a8-74c6fba40338", "node_type": "1", "metadata": {}, "hash": "b79afdf59a8853970e405a3d57258838df60ebabbe0c5c0d2f763efa99370f8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38857, "end_char_idx": 41017, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">When to use RAG</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Good for  ✅</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Not good for  ❌</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Introducing new information to the model</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Teaching ...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.43</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Data preparation: augmenting content</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What does “Augmentingcontent” mean?</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Augmenti...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "6e641843-1d7f-417e-a3a8-74c6fba40338", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e085b01-14c8-4cd5-9ab7-8d71c72e6b75", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "213541a5bef9d3eeef15c601bea3c2c786450e94899bd489b5d69bc3956754c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52779bbc-d4e7-4990-aef4-776522f6192a", "node_type": "1", "metadata": {}, "hash": "a37011a59e753a8bd76b2629edb02a625e2ecb35b443836a67aeeb82905146af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39886, "end_char_idx": 42238, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.43</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Data preparation: augmenting content</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What does “Augmentingcontent” mean?</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Augmenti...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">Yes, you can introduce new knowledge in your LLM app using RAG by retrieving information from a knowledge base or </span>
<span style="color: #00875f; text-decoration-color: #00875f">external sources to augment the model's knowledge and generate outputs relevant to the queries posed.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> How many examples do I need to fine-tune a model?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>


key: _node_content
string_value:
{"id_": "52779bbc-d4e7-4990-aef4-776522f6192a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e641843-1d7f-417e-a3a8-74c6fba40338", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "bbcbcb6a4466d7fbbb386cf5c28e00c48e5306cc59ba1e56b4b193a083bd011d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aad38009-3d55-4ee2-856d-68d6dfb26005", "node_type": "1", "metadata": {}, "hash": "914aac184da76168151c5f196379d55ca66060490f4b231b01ea31cb7426e2f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41011, "end_char_idx": 43499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">Yes, you can introduce new knowledge in your LLM app using RAG by retrieving information from a knowledge base or </span>
<span style="color: #00875f; text-decoration-color: #00875f">external sources to augment the model's knowledge and generate outputs relevant to the queries posed.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> How many examples do I need to fine-tune a model?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.68</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What is Fine-tuning</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Public Model</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Training data</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Training</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tunedmodel</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning a model consists...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "aad38009-3d55-4ee2-856d-68d6dfb26005", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52779bbc-d4e7-4990-aef4-776522f6192a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "5014e2031e7c8693cbadd44f2b9beebe56ed778913e3f71b4e2496d3b29ebf56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "052512d5-6068-4217-a23d-31fcc3950e5e", "node_type": "1", "metadata": {}, "hash": "5f6fe07088436f8c23e3b357fde7ec514c72b794553450952763bc7d44ed6c4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42600, "end_char_idx": 44624, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What is Fine-tuning</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Public Model</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Training data</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Training</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tunedmodel</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning a model consists...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.62</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">When to fine-tune</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning is good for:</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">- Following a given format or tone for the output</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">- Proce...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "052512d5-6068-4217-a23d-31fcc3950e5e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aad38009-3d55-4ee2-856d-68d6dfb26005", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "363ab8556764113ba6a9b3cf834b8c661c821ed88509b730eb7c6eeb4f0c5a2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fce1d88d-b47d-4a4b-8b60-9bfe11def6cd", "node_type": "1", "metadata": {}, "hash": "813a2099892ca6bd73f0843192ae159e6c4a25eab17eb08d9e60716b336f19f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43493, "end_char_idx": 45613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.62</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">When to fine-tune</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning is good for:</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">- Following a given format or tone for the output</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">- Proce...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.57</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Overview</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning involves adjusting theparameters of pre-trained models on aspeciﬁc dataset or t...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "fce1d88d-b47d-4a4b-8b60-9bfe11def6cd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "052512d5-6068-4217-a23d-31fcc3950e5e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9c5f1019b49f1822fb10a306e629f8c1a74a574105fd26ee9a8d7371ba4c115b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b77d1b5a-00d1-4a58-9cae-6e5fe7e7f2a1", "node_type": "1", "metadata": {}, "hash": "8bc9fa8ce34c4b65204984a98a574fe7924cdc583107ab9f09cf33e9683bf5f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 44618, "end_char_idx": 47112, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.57</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Overview</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Fine-tuning involves adjusting theparameters of pre-trained models on aspeciﬁc dataset or t...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">We recommend using </span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">50</span><span style="color: #00875f; text-decoration-color: #00875f">-</span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">100</span><span style="color: #00875f; text-decoration-color: #00875f"> examples for fine-tuning a model, even though the minimum is </span><span style="color: #00875f; text-decoration-color: #00875f; font-weight: bold">10</span><span style="color: #00875f; text-decoration-color: #00875f">.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Which metric can I use to evaluate a summarization task?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>


key: _node_content
string_value:
{"id_": "b77d1b5a-00d1-4a58-9cae-6e5fe7e7f2a1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fce1d88d-b47d-4a4b-8b60-9bfe11def6cd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "63fb60496ffb44f8e08f1d4d2b868438dad9919a8bca90250ce207e778f99786", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed0ae047-4211-4c98-aa06-6dae19a05ea1", "node_type": "1", "metadata": {}, "hash": "23b24f5646d126a18d73f7e318fad42e04cd46b401b6a23d47e75eb3067331cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 46489, "end_char_idx": 48526, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Which metric can I use to evaluate a summarization task?</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.53</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">ROUGE is a common metric for evaluating machine summariz...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.49</span>
</pre>


key: _node_content
string_value:
{"id_": "ed0ae047-4211-4c98-aa06-6dae19a05ea1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b77d1b5a-00d1-4a58-9cae-6e5fe7e7f2a1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "24c2e83d384ad6f7d127427051c6d431e9d9bb1154a8b3a1df2a0d59a47eceb2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b0e4313-6308-451f-baf3-3963eee64d63", "node_type": "1", "metadata": {}, "hash": "37be058a33be894c3ed59e93a7b5323b801b43e93f6f91c8b87d23bcfda110df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 47474, "end_char_idx": 49498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">ROUGE is a common metric for evaluating machine summariz...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.49</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Component evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Subjective evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Compari...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "7b0e4313-6308-451f-baf3-3963eee64d63", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed0ae047-4211-4c98-aa06-6dae19a05ea1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ef7fad780fe6eb5a79159e4a384b2f6c417da0ad215b585bc73507ecd6268a1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b7b149f-3cfc-41d1-9150-84f330cdf6ca", "node_type": "1", "metadata": {}, "hash": "b7b0386d7119179de071a83afe5d53fc654116e61ea983d1f66847886f79e47d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 48531, "end_char_idx": 50555, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Component evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Subjective evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Compari...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.48</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">BLEU score is another standard metric, this time focusin...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "9b7b149f-3cfc-41d1-9150-84f330cdf6ca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b0e4313-6308-451f-baf3-3963eee64d63", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f63b7bc768b8e2f26e5af98fb8edf43bd431f0708378246da710cebb10ac015c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "af113ea2-5342-4b78-96dc-2f1b7fbb5f48", "node_type": "1", "metadata": {}, "hash": "d64ff46d20f88f1726d61f508953d8c11f8b54905fa9d997c4ea17018821e4c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49492, "end_char_idx": 51703, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.48</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">BLEU score is another standard metric, this time focusin...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">ROUGE is a common metric you can use to evaluate a summarization task.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Give me a detailed example for an evaluation process where we are looking for a clear answer to compare to a</span>
<span style="color: #af005f; text-decoration-color: #af005f">ground truth.</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>


key: _node_content
string_value:
{"id_": "af113ea2-5342-4b78-96dc-2f1b7fbb5f48", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b7b149f-3cfc-41d1-9150-84f330cdf6ca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "40ef5047b37a9e02a74c6cea8157ad2d72b25f5258f4525bf65e8f7936c750b0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5a802d5-f330-4f89-b66d-5ef6ed7fa261", "node_type": "1", "metadata": {}, "hash": "21ae55db8ab4fe75d8d4c228c3166894aceaca747fd5523c99f1f54151f08420", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 50549, "end_char_idx": 52828, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">ROUGE is a common metric you can use to evaluate a summarization task.</span>

--------------


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #af005f; text-decoration-color: #af005f; font-weight: bold">QUERY:</span><span style="color: #af005f; text-decoration-color: #af005f"> Give me a detailed example for an evaluation process where we are looking for a clear answer to compare to a</span>
<span style="color: #af005f; text-decoration-color: #af005f">ground truth.</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">Matching content:</span>

</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.60</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What are evals</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Example</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Our ground truth matches the predicted answer, so the evaluation passes!</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Eval...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "f5a802d5-f330-4f89-b66d-5ef6ed7fa261", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "af113ea2-5342-4b78-96dc-2f1b7fbb5f48", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f5dfb3d58595a110f427641fb2d4310a2eb5fae16338cdcec127015844f4c6e1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60ebf99b-f6f6-45b7-abd3-a585e0f1a269", "node_type": "1", "metadata": {}, "hash": "390f2232ed3c77cd6474f0ef027e27972a09866bb451bc2ceade6168bd1fd7fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51708, "end_char_idx": 53885, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.60</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What are evals</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Example</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Our ground truth matches the predicted answer, so the evaluation passes!</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Eval...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.59</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What are evals</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Example</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">An evaluation contains a question and a correct answer. We call this the grou...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "60ebf99b-f6f6-45b7-abd3-a585e0f1a269", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5a802d5-f330-4f89-b66d-5ef6ed7fa261", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b50b02272eabcf899393e855bccad4eb09d438db9a66174aad4635f111fc54ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b4633451-b834-4d23-81c5-145b06e4c0b1", "node_type": "1", "metadata": {}, "hash": "ee47e9d7e2d52f10d301ada39b25dee618d51c492b9037c5df1b728a76863c70", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 52822, "end_char_idx": 54242, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.59</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What are evals</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Example</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">An evaluation contains a question and a correct answer. We call this the grou...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>


key: _node_content
string_value:
{"id_": "b4633451-b834-4d23-81c5-145b06e4c0b1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60ebf99b-f6f6-45b7-abd3-a585e0f1a269", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9c65e6e64cfa3fddff40575d2928fda39a4cbe8ec98b4560c8687797a786c2a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21755151-9747-4f4d-aaa8-6f3a2140765a", "node_type": "1", "metadata": {}, "hash": "5e2796b03cc5e9a22146d789ca69b93e8a03a016d867cad6ab570ddb10c24641", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53190, "end_char_idx": 55214, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What are evals</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Example</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">An evaluation contains a question and a correct answer. We call this the grou...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-style: italic">Similarity: </span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold; font-style: italic">0.50</span>
</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What they’re good for</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What to be aware of</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">A good sta...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>


key: _node_content
string_value:
{"id_": "21755151-9747-4f4d-aaa8-6f3a2140765a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b4633451-b834-4d23-81c5-145b06e4c0b1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7d31b67a34fd6de047b1bb974fc5412bed243344eeb3d70e1925865885e6450a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8aa0682d-4b5f-4485-b8b1-9988f472c966", "node_type": "1", "metadata": {}, "hash": "2d613399576b84432d72fe3e3edb284de0bc4ac7d4badf03828a130325b0468c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 54247, "end_char_idx": 57652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Technical patterns</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">Metric-based evaluations</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What they’re good for</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">What to be aware of</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">●</span>
<span style="color: #5f5f5f; text-decoration-color: #5f5f5f">A good sta...</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">[</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f">/</span><span style="color: #5f5f5f; text-decoration-color: #5f5f5f; font-weight: bold">]</span>


</pre>




<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #008787; text-decoration-color: #008787; font-weight: bold">REPLY:</span>

<span style="color: #00875f; text-decoration-color: #00875f">The content provided is relevant and offers a detailed example for an evaluation process comparing to a ground </span>
<span style="color: #00875f; text-decoration-color: #00875f">truth. Here's a concise explanation based on the content:</span>

<span style="color: #00875f; text-decoration-color: #00875f">In the given example, the evaluation process involves a question-and-answer scenario to verify the accuracy of </span>
<span style="color: #00875f; text-decoration-color: #00875f">information retrieved by a tool or system in response to a query. The question posed is, </span><span style="color: #00875f; text-decoration-color: #00875f">"What is the population of</span>
<span style="color: #00875f; text-decoration-color: #00875f">Canada?"</span><span style="color: #00875f; text-decoration-color: #00875f"> The ground truth, or the correct answer, is established as </span><span style="color: #00875f; text-decoration-color: #00875f">"The population of Canada in 2023 is 39,566,248</span>
<span style="color: #00875f; text-decoration-color: #00875f">people."</span><span style="color: #00875f; text-decoration-color: #00875f"> A tool labeled </span><span style="color: #00875f; text-decoration-color: #00875f">"LLM"</span><span style="color: #00875f; text-decoration-color: #00875f"> is then used to search for the answer, which predicts </span><span style="color: #00875f; text-decoration-color: #00875f">"The current population of </span>
<span style="color: #00875f; text-decoration-color: #00875f">Canada is 39,566,248 as of Tuesday, May 23, 2023."</span><span style="color: #00875f; text-decoration-color: #00875f"> This predicted answer matches the ground truth exactly, </span>
<span style="color: #00875f; text-decoration-color: #00875f">indicating that the evaluation passes. This process demonstrates how an evaluation can be used to verify the </span>
<span style="color: #00875f; text-decoration-color: #00875f">accuracy of information retrieved by a tool, comparing the predicted answer to the ground truth to ensure </span>
<span style="color: #00875f; text-decoration-color: #00875f">correctness.</span>

--------------


key: _node_content
string_value:
{"id_": "8aa0682d-4b5f-4485-b8b1-9988f472c966", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ca68ea7-724d-4fe9-80e8-3c74fbd2c09f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0dfce4284c29148170f2773c9757bb61d730eadd741d4da27f0bb06eb617c262", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21755151-9747-4f4d-aaa8-6f3a2140765a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Parse_PDF_docs_for_RAG.ipynb", "title": "Parsing PDF documents for RAG applications", "tokens": 17715, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b4b4f597f6dcada7d88a93fe6ea4586fcf3a5461802cafe318c5f268fb2335e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 57655, "end_char_idx": 58851, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</pre>



## Wrapping up

In this notebook, we have learned how to develop a basic RAG pipeline based on PDF documents. This includes:

- How to parse pdf documents, taking slide decks and an export from an HTML page as examples, using a python library as well as GPT-4V to interpret the visuals
- How to process the extracted content, clean it and chunk it into several pieces
- How to embed the processed content using OpenAI embeddings
- How to retrieve content that is relevant to an input query
- How to use GPT-4-turbo to generate an answer using the retrieved content as context

If you want to explore further, consider these optimisations:

- Playing around with the prompts provided as examples
- Chunking the content further and adding metadata as context to each chunk
- Adding rule-based filtering on the retrieval results or re-ranking results to surface to most relevant content

You can apply the techniques covered in this notebook to multiple use cases, such as assistants that can access your proprietary data, customer service or FAQ bots that can read from your internal policies, or anything that requires leveraging rich documents that would be better understood as images.


key: _node_content
string_value:
{"id_": "d7857a65-ed80-4127-96cc-c2bd3ea9e9c0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "420f1988-c895-4824-a13c-9625590092af", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "688e85a182ab5906a452b7f70e8aa5779cb8855d53fc2b1665272f11f888318b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90edb530-97e6-45e9-9a6b-940d243334f7", "node_type": "1", "metadata": {}, "hash": "ceff499f9b12eb7e39fc76d174c27b66b86100964ea46e7dd7a7b3bcec56cba4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3493, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Code search using embeddings

This notebook shows how Ada embeddings can be used to implement semantic code search. For this demonstration, we use our own [openai-python code repository](https://github.com/openai/openai-python). We implement a simple version of file parsing and extracting of functions from python files, which can be embedded, indexed, and queried.

### Helper Functions

We first setup some simple parsing functions that allow us to extract important information from our codebase.


```python
import pandas as pd
from pathlib import Path

DEF_PREFIXES = ['def ', 'async def ']
NEWLINE = '\n'

def get_function_name(code):
    """
    Extract function name from a line beginning with 'def' or 'async def'.
    """
    for prefix in DEF_PREFIXES:
        if code.startswith(prefix):
            return code[len(prefix): code.index('(')]


def get_until_no_space(all_lines, i):
    """
    Get all lines until a line outside the function definition is found.
    """
    ret = [all_lines[i]]
    for j in range(i + 1, len(all_lines)):
        if len(all_lines[j]) == 0 or all_lines[j][0] in [' ', '\t', ')']:
            ret.append(all_lines[j])
        else:
            break
    return NEWLINE.join(ret)


def get_functions(filepath):
    """
    Get all functions in a Python file.
    """
    with open(filepath, 'r') as file:
        all_lines = file.read().replace('\r', NEWLINE).split(NEWLINE)
        for i, l in enumerate(all_lines):
            for prefix in DEF_PREFIXES:
                if l.startswith(prefix):
                    code = get_until_no_space(all_lines, i)
                    function_name = get_function_name(code)
                    yield {
                        'code': code,
                        'function_name': function_name,
                        'filepath': filepath,
                    }
                    break


def extract_functions_from_repo(code_root):
    """
    Extract all .py functions from the repository.
    """
    code_files = list(code_root.glob('**/*.py'))

    num_files = len(code_files)
    print(f'Total number of .py files: {num_files}')

    if num_files == 0:
        print('Verify openai-python repo exists and code_root is set correctly.')
        return None

    all_funcs = [
        func
        for code_file in code_files
        for func in get_functions(str(code_file))
    ]

    num_funcs = len(all_funcs)
    print(f'Total number of functions extracted: {num_funcs}')

    return all_funcs
```

# Data Loading

We'll first load the openai-python folder and extract the needed information using the functions we defined above.


```python
# Set user root directory to the 'openai-python' repository
root_dir = Path.home()

# Assumes the 'openai-python' repository exists in the user's root directory
code_root = root_dir / 'openai-python'

# Extract all functions from the repository
all_funcs = extract_functions_from_repo(code_root)
```

    Total number of .py files: 51
    Total number of functions extracted: 97


Now that we have our content, we can pass the data to the `text-embedding-3-small` model and get back our vector embeddings.


```python
from utils.embeddings_utils import get_embedding

df = pd.DataFrame(all_funcs)
df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, model='text-embedding-3-small'))
df['filepath'] = df['filepath'].map(lambda x: Path(x).relative_to(code_root))
df.to_csv("data/code_search_openai-python.csv", index=False)
df.head()
```


key: _node_content
string_value:
{"id_": "90edb530-97e6-45e9-9a6b-940d243334f7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "420f1988-c895-4824-a13c-9625590092af", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "688e85a182ab5906a452b7f70e8aa5779cb8855d53fc2b1665272f11f888318b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7857a65-ed80-4127-96cc-c2bd3ea9e9c0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ef97641f1ea16b5b7fc1a47a254b987e43b2d8560b65786f08a85055e716bb54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92e959e5-1997-4c13-b4dc-0a3e003a5186", "node_type": "1", "metadata": {}, "hash": "772462d4fab00bfd7cb7d9837b5a09685351944f0bc8a3dfbfb84c11d67b1283", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1883, "end_char_idx": 5101, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def extract_functions_from_repo(code_root):
    """
    Extract all .py functions from the repository.
    """
    code_files = list(code_root.glob('**/*.py'))

    num_files = len(code_files)
    print(f'Total number of .py files: {num_files}')

    if num_files == 0:
        print('Verify openai-python repo exists and code_root is set correctly.')
        return None

    all_funcs = [
        func
        for code_file in code_files
        for func in get_functions(str(code_file))
    ]

    num_funcs = len(all_funcs)
    print(f'Total number of functions extracted: {num_funcs}')

    return all_funcs
```

# Data Loading

We'll first load the openai-python folder and extract the needed information using the functions we defined above.


```python
# Set user root directory to the 'openai-python' repository
root_dir = Path.home()

# Assumes the 'openai-python' repository exists in the user's root directory
code_root = root_dir / 'openai-python'

# Extract all functions from the repository
all_funcs = extract_functions_from_repo(code_root)
```

    Total number of .py files: 51
    Total number of functions extracted: 97


Now that we have our content, we can pass the data to the `text-embedding-3-small` model and get back our vector embeddings.


```python
from utils.embeddings_utils import get_embedding

df = pd.DataFrame(all_funcs)
df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, model='text-embedding-3-small'))
df['filepath'] = df['filepath'].map(lambda x: Path(x).relative_to(code_root))
df.to_csv("data/code_search_openai-python.csv", index=False)
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>code</th>
      <th>function_name</th>
      <th>filepath</th>
      <th>code_embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>def _console_log_level():\n    if openai.log i...</td>
      <td>_console_log_level</td>
      <td>openai/util.py</td>
      <td>[0.005937571171671152, 0.05450401455163956, 0....</td>
    </tr>
    <tr>
      <th>1</th>
      <td>def log_debug(message, **params):\n    msg = l...</td>
      <td>log_debug</td>
      <td>openai/util.py</td>
      <td>[0.017557814717292786, 0.05647840350866318, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>def log_info(message, **params):\n    msg = lo...</td>
      <td>log_info</td>
      <td>openai/util.py</td>
      <td>[0.022524144500494003, 0.06219055876135826, -0...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>def log_warn(message, **params):\n    msg = lo...</td>
      <td>log_warn</td>
      <td>openai/util.py</td>
      <td>[0.030524108558893204, 0.0667714849114418, -0....</td>
    </tr>
    <tr>
      <th>4</th>
      <td>def logfmt(props):\n    def fmt(key, val):\n  ...</td>
      <td>logfmt</td>
      <td>openai/util.py</td>
      <td>[0.05337328091263771, 0.03697286546230316, -0....</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "92e959e5-1997-4c13-b4dc-0a3e003a5186", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "420f1988-c895-4824-a13c-9625590092af", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "688e85a182ab5906a452b7f70e8aa5779cb8855d53fc2b1665272f11f888318b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90edb530-97e6-45e9-9a6b-940d243334f7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ebe7f9c8d6d2e8c4edd4ea2670d00e8fee611a185efa88109c3f283833a3977d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ead8626-c853-40ac-896d-f727afce223e", "node_type": "1", "metadata": {}, "hash": "6b4275a7284796c3380c98c21c2edf7d70f67550d62a280412fa549c0657d8c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5105, "end_char_idx": 7982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Testing

Let's test our endpoint with some simple queries. If you're familiar with the `openai-python` repository, you'll see that we're able to easily find functions we're looking for only a simple English description.

We define a search_functions method that takes our data that contains our embeddings, a query string, and some other configuration options. The process of searching our database works like such:

1. We first embed our query string (code_query) with `text-embedding-3-small`. The reasoning here is that a query string like 'a function that reverses a string' and a function like 'def reverse(string): return string[::-1]' will be very similar when embedded.
2. We then calculate the cosine similarity between our query string embedding and all data points in our database. This gives a distance between each point and our query.
3. We finally sort all of our data points by their distance to our query string and return the number of results requested in the function parameters. 


```python
from utils.embeddings_utils import cosine_similarity

def search_functions(df, code_query, n=3, pprint=True, n_lines=7):
    embedding = get_embedding(code_query, model='text-embedding-3-small')
    df['similarities'] = df.code_embedding.apply(lambda x: cosine_similarity(x, embedding))

    res = df.sort_values('similarities', ascending=False).head(n)

    if pprint:
        for r in res.iterrows():
            print(f"{r[1].filepath}:{r[1].function_name}  score={round(r[1].similarities, 3)}")
            print("\n".join(r[1].code.split("\n")[:n_lines]))
            print('-' * 70)

    return res
```


```python
res = search_functions(df, 'fine-tuning input data validation logic', n=3)
```

    openai/validators.py:format_inferrer_validator  score=0.453
    def format_inferrer_validator(df):
        """
        This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification.
        It will also suggest to use ada and explain train/validation split benefits.
        """
        ft_type = infer_task_type(df)
        immediate_msg = None
    ----------------------------------------------------------------------
    openai/validators.py:infer_task_type  score=0.37
    def infer_task_type(df):
        """
        Infer the likely fine-tuning task type from the data
        """
        CLASSIFICATION_THRESHOLD = 3  # min_average instances of each class
        if sum(df.prompt.str.len()) == 0:
            return "open-ended generation"
    ----------------------------------------------------------------------
    openai/validators.py:apply_validators  score=0.369
    def apply_validators(
        df,
        fname,
        remediation,
        validators,
        auto_accept,
        write_out_file_func,
    ----------------------------------------------------------------------


key: _node_content
string_value:
{"id_": "1ead8626-c853-40ac-896d-f727afce223e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "420f1988-c895-4824-a13c-9625590092af", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "688e85a182ab5906a452b7f70e8aa5779cb8855d53fc2b1665272f11f888318b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92e959e5-1997-4c13-b4dc-0a3e003a5186", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Code_search_using_embeddings.ipynb", "title": "Data Loading", "tokens": 2354, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a9488a0b21b9d0347388e2f85be1444049d686edf5028057047910065f496a28", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6733, "end_char_idx": 9958, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
res = search_functions(df, 'fine-tuning input data validation logic', n=3)
```

    openai/validators.py:format_inferrer_validator  score=0.453
    def format_inferrer_validator(df):
        """
        This validator will infer the likely fine-tuning format of the data, and display it to the user if it is classification.
        It will also suggest to use ada and explain train/validation split benefits.
        """
        ft_type = infer_task_type(df)
        immediate_msg = None
    ----------------------------------------------------------------------
    openai/validators.py:infer_task_type  score=0.37
    def infer_task_type(df):
        """
        Infer the likely fine-tuning task type from the data
        """
        CLASSIFICATION_THRESHOLD = 3  # min_average instances of each class
        if sum(df.prompt.str.len()) == 0:
            return "open-ended generation"
    ----------------------------------------------------------------------
    openai/validators.py:apply_validators  score=0.369
    def apply_validators(
        df,
        fname,
        remediation,
        validators,
        auto_accept,
        write_out_file_func,
    ----------------------------------------------------------------------



```python
res = search_functions(df, 'find common suffix', n=2, n_lines=10)
```

    openai/validators.py:get_common_xfix  score=0.487
    def get_common_xfix(series, xfix="suffix"):
        """
        Finds the longest common suffix or prefix of all the values in a series
        """
        common_xfix = ""
        while True:
            common_xfixes = (
                series.str[-(len(common_xfix) + 1) :]
                if xfix == "suffix"
                else series.str[: len(common_xfix) + 1]
    ----------------------------------------------------------------------
    openai/validators.py:common_completion_suffix_validator  score=0.449
    def common_completion_suffix_validator(df):
        """
        This validator will suggest to add a common suffix to the completion if one doesn't already exist in case of classification or conditional generation.
        """
        error_msg = None
        immediate_msg = None
        optional_msg = None
        optional_fn = None
    
        ft_type = infer_task_type(df)
    ----------------------------------------------------------------------



```python
res = search_functions(df, 'Command line interface for fine-tuning', n=1, n_lines=20)
```

    openai/cli.py:tools_register  score=0.391
    def tools_register(parser):
        subparsers = parser.add_subparsers(
            title="Tools", help="Convenience client side tools"
        )
    
        def help(args):
            parser.print_help()
    
        parser.set_defaults(func=help)
    
        sub = subparsers.add_parser("fine_tunes.prepare_data")
        sub.add_argument(
            "-f",
            "--file",
            required=True,
            help="JSONL, JSON, CSV, TSV, TXT or XLSX file containing prompt-completion examples to be analyzed."
            "This should be the local file path.",
        )
        sub.add_argument(
            "-q",
    ----------------------------------------------------------------------


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb


key: _node_content
string_value:
{"id_": "5eabf698-3886-4179-a741-154d9881f217", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b4d6f57c-86a9-4193-8edc-e717303eb0e4", "node_type": "1", "metadata": {}, "hash": "eb3e2b6df99bc6e6a26ace6c032b31afe81794e5040149be68e856459919748e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3265, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Named Entity Recognition (NER) to Enrich Text

`Named Entity Recognition` (NER) is a `Natural Language Processing` task that identifies and classifies named entities (NE) into predefined semantic categories (such as persons, organizations, locations, events, time expressions, and quantities). By converting raw text into structured information, NER makes data more actionable, facilitating tasks like information extraction, data aggregation, analytics, and social media monitoring.

This notebook demonstrates how to carry out NER with [chat completion](https://platform.openai.com/docs/api-reference/chat) and [functions-calling](https://platform.openai.com/docs/guides/gpt/function-calling) to enrich a text with links to a knowledge base such as Wikipedia:

**Text:**

*In Germany, in 1440, goldsmith Johannes Gutenberg invented the movable-type printing press. His work led to an information revolution and the unprecedented mass-spread of literature throughout Europe. Modelled on the design of the existing screw presses, a single Renaissance movable-type printing press could produce up to 3,600 pages per workday.*

**Text enriched with Wikipedia links:**

*In [Germany](https://en.wikipedia.org/wiki/Germany), in 1440, goldsmith [Johannes Gutenberg]() invented the [movable-type printing press](https://en.wikipedia.org/wiki/Movable_Type). His work led to an [information revolution](https://en.wikipedia.org/wiki/Information_revolution) and the unprecedented mass-spread of literature throughout [Europe](https://en.wikipedia.org/wiki/Europe). Modelled on the design of the existing screw presses, a single [Renaissance](https://en.wikipedia.org/wiki/Renaissance) [movable-type printing press](https://en.wikipedia.org/wiki/Movable_Type) could produce up to 3,600 pages per workday.*

**Inference Costs:** The notebook also illustrates how to estimate OpenAI API costs.

### 1. Setup

#### 1.1 Install/Upgrade Python packages


```python
%pip install --upgrade openai --quiet
%pip install --upgrade nlpia2-wikipedia --quiet
%pip install --upgrade tenacity --quiet
```

    Note: you may need to restart the kernel to use updated packages.
    Note: you may need to restart the kernel to use updated packages.
    Note: you may need to restart the kernel to use updated packages.


#### 1.2 Load packages and OPENAI_API_KEY

You can generate an API key in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details.

This notebook works with the latest OpeanAI models `gpt-3.5-turbo-0613` and `gpt-4-0613`.


```python
import json
import logging
import os

import openai
import wikipedia

from typing import Optional
from IPython.display import display, Markdown
from tenacity import retry, wait_random_exponential, stop_after_attempt

logging.basicConfig(level=logging.INFO, format=' %(asctime)s - %(levelname)s - %(message)s')

OPENAI_MODEL = 'gpt-3.5-turbo-0613'

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 2. Define the NER labels to be Identified

We define a standard set of NER labels to showcase a wide range of use cases. However, for our specific task of enriching text with knowledge base links, only a subset is practically required.


key: _node_content
string_value:
{"id_": "b4d6f57c-86a9-4193-8edc-e717303eb0e4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5eabf698-3886-4179-a741-154d9881f217", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9d5720696499d0ad1d5819fd9d926ed42f9e11f63b354c853207352600cb306e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "275f8abd-5b99-4e7d-882e-3c2a1bb6abd1", "node_type": "1", "metadata": {}, "hash": "4126aa6cf3da163ac65f4fe76981ffa7e1fbf9a10ddb43bd9d67518b61748c1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1943, "end_char_idx": 5220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
%pip install --upgrade openai --quiet
%pip install --upgrade nlpia2-wikipedia --quiet
%pip install --upgrade tenacity --quiet
```

    Note: you may need to restart the kernel to use updated packages.
    Note: you may need to restart the kernel to use updated packages.
    Note: you may need to restart the kernel to use updated packages.


#### 1.2 Load packages and OPENAI_API_KEY

You can generate an API key in the OpenAI web interface. See https://platform.openai.com/account/api-keys for details.

This notebook works with the latest OpeanAI models `gpt-3.5-turbo-0613` and `gpt-4-0613`.


```python
import json
import logging
import os

import openai
import wikipedia

from typing import Optional
from IPython.display import display, Markdown
from tenacity import retry, wait_random_exponential, stop_after_attempt

logging.basicConfig(level=logging.INFO, format=' %(asctime)s - %(levelname)s - %(message)s')

OPENAI_MODEL = 'gpt-3.5-turbo-0613'

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 2. Define the NER labels to be Identified

We define a standard set of NER labels to showcase a wide range of use cases. However, for our specific task of enriching text with knowledge base links, only a subset is practically required.


```python
labels = [
    "person",      # people, including fictional characters
    "fac",         # buildings, airports, highways, bridges
    "org",         # organizations, companies, agencies, institutions
    "gpe",         # geopolitical entities like countries, cities, states
    "loc",         # non-gpe locations
    "product",     # vehicles, foods, appareal, appliances, software, toys 
    "event",       # named sports, scientific milestones, historical events
    "work_of_art", # titles of books, songs, movies
    "law",         # named laws, acts, or legislations
    "language",    # any named language
    "date",        # absolute or relative dates or periods
    "time",        # time units smaller than a day
    "percent",     # percentage (e.g., "twenty percent", "18%")
    "money",       # monetary values, including unit
    "quantity",    # measurements, e.g., weight or distance
]
```

### 3. Prepare messages

The [chat completions API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) takes a list of messages as input and delivers a model-generated message as an output. While the chat format is primarily designed for facilitating multi-turn conversations, it is equally efficient for single-turn tasks without any preceding conversation. For our purposes, we will specify a message for the system, assistant, and user roles.

#### 3.1 System Message

The `system message` (prompt) sets the assistant's behavior by defining its desired persona and task. We also delineate the specific set of entity labels we aim to identify.

Although one can instruct the model to format its response, it has to be noted that both `gpt-3.5-turbo-0613` and `gpt-4-0613` have been fine-tuned to discern when a function should be invoked, and to reply with `JSON` formatted according to the function's signature. This capability streamlines our prompt and enables us to receive structured data directly from the model.


key: _node_content
string_value:
{"id_": "275f8abd-5b99-4e7d-882e-3c2a1bb6abd1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b4d6f57c-86a9-4193-8edc-e717303eb0e4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ef688776a475bfef256974b04287959c65734f99b40e9a1977f43c6667896d7d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df566d33-4203-412f-8043-5086e5b5534f", "node_type": "1", "metadata": {}, "hash": "36f625d5bbeadd68399cc4a41637eea392c7e93387bb5761832d3d0f9a703bf0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5223, "end_char_idx": 8183, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def system_message(labels):
    return f"""
You are an expert in Natural Language Processing. Your task is to identify common Named Entities (NER) in a given text.
The possible common Named Entities (NER) types are exclusively: ({", ".join(labels)})."""
```

#### 3.2 Assistant Message

`Assistant messages` usually store previous assistant responses. However, as in our scenario, they can also be crafted to provide examples of the desired behavior. While OpenAI is able to execute `zero-shot` Named Entity Recognition, we have found that a `one-shot` approach produces more precise results.


```python
def assisstant_message():
    return f"""
EXAMPLE:
    Text: 'In Germany, in 1440, goldsmith Johannes Gutenberg invented the movable-type printing press. His work led to an information revolution and the unprecedented mass-spread / 
    of literature throughout Europe. Modelled on the design of the existing screw presses, a single Renaissance movable-type printing press could produce up to 3,600 pages per workday.'
    {{
        "gpe": ["Germany", "Europe"],
        "date": ["1440"],
        "person": ["Johannes Gutenberg"],
        "product": ["movable-type printing press"],
        "event": ["Renaissance"],
        "quantity": ["3,600 pages"],
        "time": ["workday"]
    }}
--"""
```

#### 3.3 User Message

The `user message` provides the specific text for the assistant task:


```python
def user_message(text):
    return f"""
TASK:
    Text: {text}
"""
```

### 4. OpenAI Functions (and Utils)

In an OpenAI API call, we can describe `functions` to `gpt-3.5-turbo-0613` and `gpt-4-0613` and have the model intelligently choose to output a `JSON` object containing arguments to call those `functions`. It's important to note that the [chat completions API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) doesn't actually execute the `function`. Instead, it provides the `JSON` output, which can then be used to call the `function` in our code. For more details, refer to the [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling).

Our function, `enrich_entities(text, label_entities)` gets a block of text and a dictionary containing identified labels and entities as parameters. It then associates the recognized entities with their corresponding links to the Wikipedia articles.


```python
@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))
def find_link(entity: str) -> Optional[str]:
    """
    Finds a Wikipedia link for a given entity.
    """
    try:
        titles = wikipedia.search(entity)
        if titles:
            # naively consider the first result as the best
            page = wikipedia.page(titles[0])
            return page.url
    except (wikipedia.exceptions.WikipediaException) as ex:
        logging.error(f'Error occurred while searching for Wikipedia link for entity {entity}: {str(ex)}')

    return None
```


key: _node_content
string_value:
{"id_": "df566d33-4203-412f-8043-5086e5b5534f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "275f8abd-5b99-4e7d-882e-3c2a1bb6abd1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4b69bf779512bfb8482e499a9bc9b9485ada7b7fc9eee27fe140d9814ea35cec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7faf9eb5-73d3-46ab-b666-2a728595f4d0", "node_type": "1", "metadata": {}, "hash": "7660c9d6abb17a41374d6eba5ce131692ac72e9cc6b32a48df624513bf3a5e99", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6634, "end_char_idx": 8605, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def user_message(text):
    return f"""
TASK:
    Text: {text}
"""
```

### 4. OpenAI Functions (and Utils)

In an OpenAI API call, we can describe `functions` to `gpt-3.5-turbo-0613` and `gpt-4-0613` and have the model intelligently choose to output a `JSON` object containing arguments to call those `functions`. It's important to note that the [chat completions API](https://platform.openai.com/docs/guides/gpt/chat-completions-api) doesn't actually execute the `function`. Instead, it provides the `JSON` output, which can then be used to call the `function` in our code. For more details, refer to the [OpenAI Function Calling Guide](https://platform.openai.com/docs/guides/function-calling).

Our function, `enrich_entities(text, label_entities)` gets a block of text and a dictionary containing identified labels and entities as parameters. It then associates the recognized entities with their corresponding links to the Wikipedia articles.


```python
@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))
def find_link(entity: str) -> Optional[str]:
    """
    Finds a Wikipedia link for a given entity.
    """
    try:
        titles = wikipedia.search(entity)
        if titles:
            # naively consider the first result as the best
            page = wikipedia.page(titles[0])
            return page.url
    except (wikipedia.exceptions.WikipediaException) as ex:
        logging.error(f'Error occurred while searching for Wikipedia link for entity {entity}: {str(ex)}')

    return None
```


```python
def find_all_links(label_entities:dict) -> dict:
    """ 
    Finds all Wikipedia links for the dictionary entities in the whitelist label list.
    """
    whitelist = ['event', 'gpe', 'org', 'person', 'product', 'work_of_art']
    
    return {e: find_link(e) for label, entities in label_entities.items() 
                            for e in entities
                            if label in whitelist}
```


key: _node_content
string_value:
{"id_": "7faf9eb5-73d3-46ab-b666-2a728595f4d0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df566d33-4203-412f-8043-5086e5b5534f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2b74c57767eaf56c04a1320cb133970305c1247749e50477c259e7a964fb8260", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c3a74b0-de91-436d-b0c0-fdb0e0a7798a", "node_type": "1", "metadata": {}, "hash": "98cd09c61cfd0ec9e2b691ae4b7af41615e139b060f18bc4a839bb995b36a977", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7595, "end_char_idx": 10365, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))
def find_link(entity: str) -> Optional[str]:
    """
    Finds a Wikipedia link for a given entity.
    """
    try:
        titles = wikipedia.search(entity)
        if titles:
            # naively consider the first result as the best
            page = wikipedia.page(titles[0])
            return page.url
    except (wikipedia.exceptions.WikipediaException) as ex:
        logging.error(f'Error occurred while searching for Wikipedia link for entity {entity}: {str(ex)}')

    return None
```


```python
def find_all_links(label_entities:dict) -> dict:
    """ 
    Finds all Wikipedia links for the dictionary entities in the whitelist label list.
    """
    whitelist = ['event', 'gpe', 'org', 'person', 'product', 'work_of_art']
    
    return {e: find_link(e) for label, entities in label_entities.items() 
                            for e in entities
                            if label in whitelist}
```


```python
def enrich_entities(text: str, label_entities: dict) -> str:
    """
    Enriches text with knowledge base links.
    """
    entity_link_dict = find_all_links(label_entities)
    logging.info(f"entity_link_dict: {entity_link_dict}")
    
    for entity, link in entity_link_dict.items():
        text = text.replace(entity, f"[{entity}]({link})")

    return text
```

### 4. ChatCompletion

As previously highlighted, `gpt-3.5-turbo-0613` and `gpt-4-0613` have been fine-tuned to detect when a `function` should to be called. Moreover, they can produce a `JSON` response that conforms to the `function` signature. Here's the sequence we follow:

1. Define our `function` and its associated `JSON` Schema.
2. Invoke the model using the `messages`, `tools` and `tool_choice` parameters.
3. Convert the output into a `JSON` object, and then call the `function` with the `arguments` provided by the model.

In practice, one might want to re-invoke the model again by appending the `function` response as a new message, and let the model summarize the results back to the user. Nevertheless, for our purposes, this step is not needed.

*Note that in a real-case scenario it is strongly recommended to build in user confirmation flows before taking actions.*

#### 4.1 Define our Function and JSON schema

Since we want the model to output a dictionary of labels and recognized entities:

```python
{   
    "gpe": ["Germany", "Europe"],   
    "date": ["1440"],   
    "person": ["Johannes Gutenberg"],   
    "product": ["movable-type printing press"],   
    "event": ["Renaissance"],   
    "quantity": ["3,600 pages"],   
    "time": ["workday"]   
}   
```
we need to define the corresponding `JSON` schema to be passed to the `tools` parameter:


key: _node_content
string_value:
{"id_": "2c3a74b0-de91-436d-b0c0-fdb0e0a7798a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7faf9eb5-73d3-46ab-b666-2a728595f4d0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "90dc0bc2632735e8e2f59599fef21ae174e48fbd3bf43195bb88fd571dd7c3a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80b9e2ec-4545-4b94-8d3f-3458d0e0fc07", "node_type": "1", "metadata": {}, "hash": "fd52f3aa36c997a614dd853d9b03ce869acc18b5f884cfec05b4c1c49ab5ecc6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10369, "end_char_idx": 12927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def generate_functions(labels: dict) -> list:
    return [
        {   
            "type": "function",
            "function": {
                "name": "enrich_entities",
                "description": "Enrich Text with Knowledge Base Links",
                "parameters": {
                    "type": "object",
                        "properties": {
                            "r'^(?:' + '|'.join({labels}) + ')$'": 
                            {
                                "type": "array",
                                "items": {
                                    "type": "string"
                                }
                            }
                        },
                        "additionalProperties": False
                },
            }
        }
    ]
```

#### 4.2 Chat Completion

Now, we invoke the model. It's important to note that we direct the API to use a specific function by setting the `tool_choice` parameter to `{"type": "function", "function" : {"name": "enrich_entities"}}`.


```python
@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))
def run_openai_task(labels, text):
    messages = [
          {"role": "system", "content": system_message(labels=labels)},
          {"role": "assistant", "content": assisstant_message()},
          {"role": "user", "content": user_message(text=text)}
      ]

    # TODO: functions and function_call are deprecated, need to be updated
    # See: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools
    response = openai.chat.completions.create(
        model="gpt-3.5-turbo-0613",
        messages=messages,
        tools=generate_functions(labels),
        tool_choice={"type": "function", "function" : {"name": "enrich_entities"}}, 
        temperature=0,
        frequency_penalty=0,
        presence_penalty=0,
    )

    response_message = response.choices[0].message
    
    available_functions = {"enrich_entities": enrich_entities}  
    function_name = response_message.tool_calls[0].function.name
    
    function_to_call = available_functions[function_name]
    logging.info(f"function_to_call: {function_to_call}")

    function_args = json.loads(response_message.tool_calls[0].function.arguments)
    logging.info(f"function_args: {function_args}")

    function_response = function_to_call(text, function_args)

    return {"model_response": response, 
            "function_response": function_response}
```

### 5. Let's Enrich a Text with Wikipedia links

#### 5.1 Run OpenAI Task


key: _node_content
string_value:
{"id_": "80b9e2ec-4545-4b94-8d3f-3458d0e0fc07", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c3a74b0-de91-436d-b0c0-fdb0e0a7798a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "5a54e06265496cefb5098b460a1e38bc68f209d0acf57faf044ee61348324be7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1bc5a35f-7e54-48a8-8ea9-7a62f0766d03", "node_type": "1", "metadata": {}, "hash": "f3612cf294d02603f04246cab52e142f369399fe139adb5c8d3869dd84023926", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11411, "end_char_idx": 14027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
@retry(wait=wait_random_exponential(min=1, max=10), stop=stop_after_attempt(5))
def run_openai_task(labels, text):
    messages = [
          {"role": "system", "content": system_message(labels=labels)},
          {"role": "assistant", "content": assisstant_message()},
          {"role": "user", "content": user_message(text=text)}
      ]

    # TODO: functions and function_call are deprecated, need to be updated
    # See: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools
    response = openai.chat.completions.create(
        model="gpt-3.5-turbo-0613",
        messages=messages,
        tools=generate_functions(labels),
        tool_choice={"type": "function", "function" : {"name": "enrich_entities"}}, 
        temperature=0,
        frequency_penalty=0,
        presence_penalty=0,
    )

    response_message = response.choices[0].message
    
    available_functions = {"enrich_entities": enrich_entities}  
    function_name = response_message.tool_calls[0].function.name
    
    function_to_call = available_functions[function_name]
    logging.info(f"function_to_call: {function_to_call}")

    function_args = json.loads(response_message.tool_calls[0].function.arguments)
    logging.info(f"function_args: {function_args}")

    function_response = function_to_call(text, function_args)

    return {"model_response": response, 
            "function_response": function_response}
```

### 5. Let's Enrich a Text with Wikipedia links

#### 5.1 Run OpenAI Task


```python
text = """The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr."""
result = run_openai_task(labels, text)
```

     2023-10-20 18:05:51,729 - INFO - function_to_call: <function enrich_entities at 0x0000021D30C462A0>
     2023-10-20 18:05:51,730 - INFO - function_args: {'person': ['John Lennon', 'Paul McCartney', 'George Harrison', 'Ringo Starr'], 'org': ['The Beatles'], 'gpe': ['Liverpool'], 'date': ['1960']}
     2023-10-20 18:06:09,858 - INFO - entity_link_dict: {'John Lennon': 'https://en.wikipedia.org/wiki/John_Lennon', 'Paul McCartney': 'https://en.wikipedia.org/wiki/Paul_McCartney', 'George Harrison': 'https://en.wikipedia.org/wiki/George_Harrison', 'Ringo Starr': 'https://en.wikipedia.org/wiki/Ringo_Starr', 'The Beatles': 'https://en.wikipedia.org/wiki/The_Beatles', 'Liverpool': 'https://en.wikipedia.org/wiki/Liverpool'}


#### 5.2 Function Response


```python
display(Markdown(f"""**Text:** {text}   
                     **Enriched_Text:** {result['function_response']}"""))
```


key: _node_content
string_value:
{"id_": "1bc5a35f-7e54-48a8-8ea9-7a62f0766d03", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93945a11-e5ea-47cd-90b6-7eda9926b463", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "578873abcfd5fffd8c0cfcd310cc018e4f9767cf2fd3c6b6abf04dfb9ea293e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80b9e2ec-4545-4b94-8d3f-3458d0e0fc07", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Named_Entity_Recognition_to_enrich_text.ipynb", "title": "estimate inference cost assuming gpt-3.5-turbo (4K context)", "tokens": 3604, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "48781839b7db74c574cea73cc553be8176df218b2b23ab8bd5ac19b83b3b2266", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12930, "end_char_idx": 15361, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
text = """The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr."""
result = run_openai_task(labels, text)
```

     2023-10-20 18:05:51,729 - INFO - function_to_call: <function enrich_entities at 0x0000021D30C462A0>
     2023-10-20 18:05:51,730 - INFO - function_args: {'person': ['John Lennon', 'Paul McCartney', 'George Harrison', 'Ringo Starr'], 'org': ['The Beatles'], 'gpe': ['Liverpool'], 'date': ['1960']}
     2023-10-20 18:06:09,858 - INFO - entity_link_dict: {'John Lennon': 'https://en.wikipedia.org/wiki/John_Lennon', 'Paul McCartney': 'https://en.wikipedia.org/wiki/Paul_McCartney', 'George Harrison': 'https://en.wikipedia.org/wiki/George_Harrison', 'Ringo Starr': 'https://en.wikipedia.org/wiki/Ringo_Starr', 'The Beatles': 'https://en.wikipedia.org/wiki/The_Beatles', 'Liverpool': 'https://en.wikipedia.org/wiki/Liverpool'}


#### 5.2 Function Response


```python
display(Markdown(f"""**Text:** {text}   
                     **Enriched_Text:** {result['function_response']}"""))
```


**Text:** The Beatles were an English rock band formed in Liverpool in 1960, comprising John Lennon, Paul McCartney, George Harrison, and Ringo Starr.   
                     **Enriched_Text:** [The Beatles](https://en.wikipedia.org/wiki/The_Beatles) were an English rock band formed in [Liverpool](https://en.wikipedia.org/wiki/Liverpool) in 1960, comprising [John Lennon](https://en.wikipedia.org/wiki/John_Lennon), [Paul McCartney](https://en.wikipedia.org/wiki/Paul_McCartney), [George Harrison](https://en.wikipedia.org/wiki/George_Harrison), and [Ringo Starr](https://en.wikipedia.org/wiki/Ringo_Starr).


#### 5.3 Token Usage

To estimate the inference costs, we can parse the response's "usage" field. Detailed token costs per model are available in the [OpenAI Pricing Guide](https://openai.com/pricing):


```python
# estimate inference cost assuming gpt-3.5-turbo (4K context)
i_tokens  = result["model_response"].usage.prompt_tokens 
o_tokens = result["model_response"].usage.completion_tokens 

i_cost = (i_tokens / 1000) * 0.0015
o_cost = (o_tokens / 1000) * 0.002

print(f"""Token Usage
    Prompt: {i_tokens} tokens
    Completion: {o_tokens} tokens
    Cost estimation: ${round(i_cost + o_cost, 5)}""")
```

    Token Usage
        Prompt: 331 tokens
        Completion: 47 tokens
        Cost estimation: $0.00059


key: _node_content
string_value:
{"id_": "cec02dba-932d-41bd-8d8c-1d52dd10b87d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/README.ipynb", "title": "Vector Databases", "tokens": 586, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b6324ffa-0965-4a49-8865-2b1a50485796", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/README.ipynb", "title": "Vector Databases", "tokens": 586, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "97b9842dbf16fff1c016b73b36580f094d3e88262a424c2fc42846ed9e5ee3f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Vector Databases

This section of the OpenAI Cookbook showcases many of the vector databases available to support your semantic search use cases.

Vector databases can be a great accompaniment for knowledge retrieval applications, which reduce hallucinations by providing the LLM with the relevant context to answer questions.

Each provider has their own named directory, with a standard notebook to introduce you to using our API with their product, and any supplementary notebooks they choose to add to showcase their functionality.

## Guides & deep dives
- [AnalyticDB](https://www.alibabacloud.com/help/en/analyticdb-for-postgresql/latest/get-started-with-analyticdb-for-postgresql)
- [Cassandra/Astra DB](https://docs.datastax.com/en/astra-serverless/docs/vector-search/qandasimsearch-quickstart.html)
- [Azure AI Search](https://learn.microsoft.com/azure/search/search-get-started-vector)
- [Azure SQL Database](https://learn.microsoft.com/azure/azure-sql/database/ai-artificial-intelligence-intelligent-applications?view=azuresql)
- [Chroma](https://docs.trychroma.com/getting-started)
- [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html)
- [Hologres](https://www.alibabacloud.com/help/en/hologres/latest/procedure-to-use-hologres)
- [Kusto](https://learn.microsoft.com/en-us/azure/data-explorer/web-query-data)
- [Milvus](https://milvus.io/docs/example_code.md)
- [MyScale](https://docs.myscale.com/en/quickstart/)
- [MongoDB](https://www.mongodb.com/products/platform/atlas-vector-search)
- [Neon Postgres](https://neon.tech/docs/ai/ai-intro)
- [Pinecone](https://docs.pinecone.io/docs/quickstart)
- [PolarDB](https://www.alibabacloud.com/help/en/polardb/latest/quick-start)
- [Qdrant](https://qdrant.tech/documentation/quick-start/)
- [Redis](https://github.com/RedisVentures/simple-vecsim-intro)
- [SingleStoreDB](https://www.singlestore.com/blog/how-to-get-started-with-singlestore/)
- [Supabase](https://supabase.com/docs/guides/ai)
- [Tembo](https://tembo.io/docs/product/stacks/ai/vectordb)
- [Typesense](https://typesense.org/docs/guide/)
- [Vespa AI](https://vespa.ai/)
- [Weaviate](https://weaviate.io/developers/weaviate/quickstart)
- [Zilliz](https://docs.zilliz.com/docs/quick-start-1)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb


key: _node_content
string_value:
{"id_": "7eaa0382-380d-41c7-a04a-4614f9184cd4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e5acffc-dfe7-476a-96f2-b6e1b5155c73", "node_type": "1", "metadata": {}, "hash": "6b41a7a857a0b428bd46502f5bc5e40174f60e462e440a2d7d51cac8d7368019", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2225, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Kusto as a Vector database for AI embeddings

This Notebook provides step by step instuctions on using Azure Data Explorer (Kusto) as a vector database with OpenAI embeddings. 

This notebook presents an end-to-end process of:

1. Using precomputed embeddings created by OpenAI API.
2. Storing the embeddings in Kusto.
3. Converting raw text query to an embedding with OpenAI API.
4. Using Kusto to perform cosine similarity search in the stored embeddings


### Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. Azure Data Explorer(Kusto) server instance. https://azure.microsoft.com/en-us/products/data-explorer
3. Azure OpenAI credentials or OpenAI API key.


```python
%pip install wget
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, -1, Finished, Available)







    Collecting wget
      Downloading wget-3.2.zip (10 kB)
      Preparing metadata (setup.py) ... [?25ldone
    [?25hBuilding wheels for collected packages: wget
      Building wheel for wget (setup.py) ... [?25l- done
    [?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=10fd8aa1d20fd49c36389dc888acc721d0578c5a0635fc9fc5dc642c0f49522e
      Stored in directory: /home/trusted-service-user/.cache/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769
    Successfully built wget
    Installing collected packages: wget
    Successfully installed wget-3.2
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;49m -> [0m[32;49m23.1.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.







    Warning: PySpark kernel has been restarted to use updated packages.
    



```python
%pip install openai
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, -1, Finished, Available)







    Collecting openai
      Downloading openai-0.27.6-py3-none-any.whl (71 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m71.9/71.9 kB[0m [31m1.


key: _node_content
string_value:
{"id_": "2e5acffc-dfe7-476a-96f2-b6e1b5155c73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7eaa0382-380d-41c7-a04a-4614f9184cd4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "abcc4cd3f700ee50ae9b8d6217a22f9bd2cf658b4e5ec0a5fa1656c97530b8ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b040b7f-fff8-4c17-958e-084b82d00b79", "node_type": "1", "metadata": {}, "hash": "9d8556a7c65a4f971d45585bf6ec0c76dbc138fe7cdd2bcc95d467074c37db4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1852, "end_char_idx": 4267, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Warning: PySpark kernel has been restarted to use updated packages.
    



```python
%pip install openai
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, -1, Finished, Available)







    Collecting openai
      Downloading openai-0.27.6-py3-none-any.whl (71 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m71.9/71.9 kB[0m [31m1.7 MB/s[0m eta [36m0:00:00[0m00:01[0m
    [?25hRequirement already satisfied: tqdm in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from openai) (4.65.0)
    Requirement already satisfied: requests>=2.20 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from openai) (2.28.2)
    Requirement already satisfied: aiohttp in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from openai) (3.8.4)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (1.26.14)
    Requirement already satisfied: certifi>=2017.4.17 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (2022.12.7)
    Requirement already satisfied: idna<4,>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: charset-normalizer<4,>=2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: attrs>=17.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (22.2.0)
    Requirement already satisfied: frozenlist>=1.1.1 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: multidict<7.0,>=4.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.8.2)
    Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (4.0.


key: _node_content
string_value:
{"id_": "6b040b7f-fff8-4c17-958e-084b82d00b79", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e5acffc-dfe7-476a-96f2-b6e1b5155c73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "96b6ad5f1a7f2ca66471de9514d8d026e0c69c9b6bd85c0da695b3b0743f6938", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9cdc4a1c-c2bd-47fb-935a-03b8aaffcfac", "node_type": "1", "metadata": {}, "hash": "13e62d288df42ea07bc9615c59b32325d0cc5c6e0da7004563f384634ae316c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2951, "end_char_idx": 5274, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4.17 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (2022.12.7)
    Requirement already satisfied: idna<4,>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: charset-normalizer<4,>=2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: attrs>=17.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (22.2.0)
    Requirement already satisfied: frozenlist>=1.1.1 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: multidict<7.0,>=4.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.8.2)
    Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (4.0.2)
    Requirement already satisfied: aiosignal>=1.1.2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    Installing collected packages: openai
    Successfully installed openai-0.27.6
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;49m -> [0m[32;49m23.1.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.







    Warning: PySpark kernel has been restarted to use updated packages.
    



```python
%pip install azure-kusto-data
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, -1, Finished, Available)







    Requirement already satisfied: azure-kusto-data in /nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/lib/python3.10/site-packages (4.


key: _node_content
string_value:
{"id_": "9cdc4a1c-c2bd-47fb-935a-03b8aaffcfac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6b040b7f-fff8-4c17-958e-084b82d00b79", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4538bfa2a00736669247cbe7b5669f9bb05aaae493a79254c6f6ee38434701a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4da93846-4407-401a-884b-541359392a26", "node_type": "1", "metadata": {}, "hash": "d263f5831585e4b0b36607b88507402a8388837623bc17963ce9d8db7da5a0de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4222, "end_char_idx": 6483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (from aiohttp->openai) (4.0.2)
    Requirement already satisfied: aiosignal>=1.1.2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    Installing collected packages: openai
    Successfully installed openai-0.27.6
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;49m -> [0m[32;49m23.1.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.







    Warning: PySpark kernel has been restarted to use updated packages.
    



```python
%pip install azure-kusto-data
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, -1, Finished, Available)







    Requirement already satisfied: azure-kusto-data in /nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/lib/python3.10/site-packages (4.1.4)
    Requirement already satisfied: msal<2,>=1.9.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.21.0)
    Requirement already satisfied: python-dateutil>=2.8.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (2.8.2)
    Requirement already satisfied: azure-core<2,>=1.11.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.26.4)
    Requirement already satisfied: requests>=2.13.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (2.28.2)
    Requirement already satisfied: ijson~=3.1 in /nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/lib/python3.10/site-packages (from azure-kusto-data) (3.2.0.post0)
    Requirement already satisfied: azure-identity<2,>=1.5.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.12.0)
    Requirement already satisfied: six>=1.11.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-core<2,>=1.11.0->azure-kusto-data) (1.16.


key: _node_content
string_value:
{"id_": "4da93846-4407-401a-884b-541359392a26", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9cdc4a1c-c2bd-47fb-935a-03b8aaffcfac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "efb09f2e681ba67a6b065bf2c66843186b97f7bdfb51c1c420fcdc6d3c8538c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed498f57-5510-4464-91e9-161ecfd28ee4", "node_type": "1", "metadata": {}, "hash": "43b6a42f3a930bf9a8feb2b0cf23aaa3b919c99fd96e082c8ab1861460c4bee2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5254, "end_char_idx": 7706, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (4.1.4)
    Requirement already satisfied: msal<2,>=1.9.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.21.0)
    Requirement already satisfied: python-dateutil>=2.8.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (2.8.2)
    Requirement already satisfied: azure-core<2,>=1.11.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.26.4)
    Requirement already satisfied: requests>=2.13.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (2.28.2)
    Requirement already satisfied: ijson~=3.1 in /nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/lib/python3.10/site-packages (from azure-kusto-data) (3.2.0.post0)
    Requirement already satisfied: azure-identity<2,>=1.5.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-kusto-data) (1.12.0)
    Requirement already satisfied: six>=1.11.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-core<2,>=1.11.0->azure-kusto-data) (1.16.0)
    Requirement already satisfied: typing-extensions>=4.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-core<2,>=1.11.0->azure-kusto-data) (4.5.0)
    Requirement already satisfied: cryptography>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-identity<2,>=1.5.0->azure-kusto-data) (40.0.1)
    Requirement already satisfied: msal-extensions<2.0.0,>=0.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-identity<2,>=1.5.0->azure-kusto-data) (1.0.0)
    Requirement already satisfied: PyJWT[crypto]<3,>=1.0.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from msal<2,>=1.9.0->azure-kusto-data) (2.6.0)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (1.26.14)
    Requirement already satisfied: charset-normalizer<4,>=2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (2.1.1)
    Requirement already satisfied: idna<4,>=2.


key: _node_content
string_value:
{"id_": "ed498f57-5510-4464-91e9-161ecfd28ee4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4da93846-4407-401a-884b-541359392a26", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "cc0bce227316003e1e1a9e9840eee49519d607142773e87b6b3203d32bf681c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2cfc81bc-6b89-496b-8428-d801f67fb6a5", "node_type": "1", "metadata": {}, "hash": "56feecf0ba36b28225d69ec59cffe2f9542b12e3538bcdc653311fb429261b63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6449, "end_char_idx": 8800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>=1.11.0->azure-kusto-data) (1.16.0)
    Requirement already satisfied: typing-extensions>=4.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-core<2,>=1.11.0->azure-kusto-data) (4.5.0)
    Requirement already satisfied: cryptography>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-identity<2,>=1.5.0->azure-kusto-data) (40.0.1)
    Requirement already satisfied: msal-extensions<2.0.0,>=0.3.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from azure-identity<2,>=1.5.0->azure-kusto-data) (1.0.0)
    Requirement already satisfied: PyJWT[crypto]<3,>=1.0.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from msal<2,>=1.9.0->azure-kusto-data) (2.6.0)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (1.26.14)
    Requirement already satisfied: charset-normalizer<4,>=2 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (3.4)
    Requirement already satisfied: certifi>=2017.4.17 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (2022.12.7)
    Requirement already satisfied: cffi>=1.12 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from cryptography>=2.5->azure-identity<2,>=1.5.0->azure-kusto-data) (1.15.1)
    Requirement already satisfied: portalocker<3,>=1.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from msal-extensions<2.0.0,>=0.3.0->azure-identity<2,>=1.5.0->azure-kusto-data) (2.7.0)
    Requirement already satisfied: pycparser in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.5->azure-identity<2,>=1.5.0->azure-kusto-data) (2.21)
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;


key: _node_content
string_value:
{"id_": "2cfc81bc-6b89-496b-8428-d801f67fb6a5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed498f57-5510-4464-91e9-161ecfd28ee4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f28494e273bb1d3b406eb487d470c23e379146da79c1dc0e97b6198b05fe1d5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41cbca6e-25fe-4ca2-b17a-5789c4a4a0ed", "node_type": "1", "metadata": {}, "hash": "6688e41007537baa45334a702bbc393a8dd0f916978bd1c220526aa8f9d3cc1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7628, "end_char_idx": 9980, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
13.0->azure-kusto-data) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (3.4)
    Requirement already satisfied: certifi>=2017.4.17 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from requests>=2.13.0->azure-kusto-data) (2022.12.7)
    Requirement already satisfied: cffi>=1.12 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from cryptography>=2.5->azure-identity<2,>=1.5.0->azure-kusto-data) (1.15.1)
    Requirement already satisfied: portalocker<3,>=1.0 in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from msal-extensions<2.0.0,>=0.3.0->azure-identity<2,>=1.5.0->azure-kusto-data) (2.7.0)
    Requirement already satisfied: pycparser in /home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.5->azure-identity<2,>=1.5.0->azure-kusto-data) (2.21)
    
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;49m -> [0m[32;49m23.1.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.







    Warning: PySpark kernel has been restarted to use updated packages.
    


### Download precomputed Embeddings



In this section we are going to load prepared embedding data, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 17, Finished, Available)





    'vector_database_wikipedia_articles_embedded.zip'




```python

import zipfile

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("/lakehouse/default/Files/data")
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 18, Finished, Available)


key: _node_content
string_value:
{"id_": "41cbca6e-25fe-4ca2-b17a-5789c4a4a0ed", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2cfc81bc-6b89-496b-8428-d801f67fb6a5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6f62f587f8cbb6b922bee25841f00b38226757bed42e8405346aca4d29fbeb61", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d09114c4-61af-41cd-b4ef-1434a944f2e0", "node_type": "1", "metadata": {}, "hash": "2f58c2dc423f858fdd4c773f02d13745663c0f07b2cfe3584e1f75b04a2ec894", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8720, "end_char_idx": 10453, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.0[0m[39;49m -> [0m[32;49m23.1.2[0m
    [1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49m/nfs4/pyenv-27214bb4-edfd-4fdd-b888-8a99075a1416/bin/python -m pip install --upgrade pip[0m
    Note: you may need to restart the kernel to use updated packages.







    Warning: PySpark kernel has been restarted to use updated packages.
    


### Download precomputed Embeddings



In this section we are going to load prepared embedding data, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 17, Finished, Available)





    'vector_database_wikipedia_articles_embedded.zip'




```python

import zipfile

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("/lakehouse/default/Files/data")
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 18, Finished, Available)



```python
import pandas as pd

from ast import literal_eval

article_df = pd.read_csv('/lakehouse/default/Files/data/vector_database_wikipedia_articles_embedded.csv')
# Read vectors from strings back into a list
article_df["title_vector"] = article_df.title_vector.apply(literal_eval)
article_df["content_vector"] = article_df.content_vector.apply(literal_eval)
article_df.head()

```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 19, Finished, Available)


key: _node_content
string_value:
{"id_": "d09114c4-61af-41cd-b4ef-1434a944f2e0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41cbca6e-25fe-4ca2-b17a-5789c4a4a0ed", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "565c8900e75d71924fe3dd663db70e3aa0bb2019ddfc5080bd1801d76bec7417", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cbbda6c5-628e-4498-9821-4bede7e5a110", "node_type": "1", "metadata": {}, "hash": "420c1c7a1a4a1c15a3e8a9dd3de40c2324ff4c2b934628f0ef2e251689866033", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9087, "end_char_idx": 11470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Warning: PySpark kernel has been restarted to use updated packages.
    


### Download precomputed Embeddings



In this section we are going to load prepared embedding data, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 17, Finished, Available)





    'vector_database_wikipedia_articles_embedded.zip'




```python

import zipfile

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("/lakehouse/default/Files/data")
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 18, Finished, Available)



```python
import pandas as pd

from ast import literal_eval

article_df = pd.read_csv('/lakehouse/default/Files/data/vector_database_wikipedia_articles_embedded.csv')
# Read vectors from strings back into a list
article_df["title_vector"] = article_df.title_vector.apply(literal_eval)
article_df["content_vector"] = article_df.content_vector.apply(literal_eval)
article_df.head()

```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 19, Finished, Available)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "cbbda6c5-628e-4498-9821-4bede7e5a110", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d09114c4-61af-41cd-b4ef-1434a944f2e0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "733f782886389c6df39ea7a303165ffb3be9c192d7a929c198f22f6c5f520d42", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e9b710e-ec54-4f67-a219-617672541402", "node_type": "1", "metadata": {}, "hash": "c92f1909b1deb45242cd42f34e51d158a18331b00df6dd18f3d4dd75bb0332c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10375, "end_char_idx": 12501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 19, Finished, Available)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.) is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere.


key: _node_content
string_value:
{"id_": "9e9b710e-ec54-4f67-a219-617672541402", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cbbda6c5-628e-4498-9821-4bede7e5a110", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6376a592a46a643ae427ebc57521ad5a12e48cc6c09fb5d8e0593c96c1465ea2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b92cdcf7-e9c6-4d59-8d89-0cf0d20e8b91", "node_type": "1", "metadata": {}, "hash": "2229932697c0e43811fa611df27097cc2a7f81050bd63cb3a479f71573e44b8c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11471, "end_char_idx": 13727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



### Store vectors in a Kusto table


Create a table & load the vectors in Kusto based on the contents in the dataframe. The spark option CreakeIfNotExists will automatically create a table if it doesn't exist



```python
# replace with your AAD Tenant ID, Kusto Cluster URI, Kusto DB name and Kusto Table
AAD_TENANT_ID = ""
KUSTO_CLUSTER =  ""
KUSTO_DATABASE = "Vector"
KUSTO_TABLE = "Wiki"
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 37, Finished, Available)



```python

kustoOptions = {"kustoCluster": KUSTO_CLUSTER, "kustoDatabase" :KUSTO_DATABASE, "kustoTable" : KUSTO_TABLE }

# Replace the auth method based on your desired authentication mechanism  - https://github.com/Azure/azure-kusto-spark/blob/master/docs/Authentication.md
access_token=mssparkutils.credentials.getToken(kustoOptions["kustoCluster"])
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 21, Finished, Available)



```python
#Pandas data frame to spark dataframe
sparkDF=spark.createDataFrame(article_df)
```


key: _node_content
string_value:
{"id_": "b92cdcf7-e9c6-4d59-8d89-0cf0d20e8b91", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e9b710e-ec54-4f67-a219-617672541402", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "304fa5c736043d190716e37aba03952cc88a57e27e67e7baf155ede656bc0cea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9706389-0a15-472c-ba62-83958c2b4e72", "node_type": "1", "metadata": {}, "hash": "948ef270714793d1c74fe9a89b7cac6865d0d6bc8d3da6ab7e6bda99903bc79c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12502, "end_char_idx": 15096, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



### Store vectors in a Kusto table


Create a table & load the vectors in Kusto based on the contents in the dataframe. The spark option CreakeIfNotExists will automatically create a table if it doesn't exist



```python
# replace with your AAD Tenant ID, Kusto Cluster URI, Kusto DB name and Kusto Table
AAD_TENANT_ID = ""
KUSTO_CLUSTER =  ""
KUSTO_DATABASE = "Vector"
KUSTO_TABLE = "Wiki"
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 37, Finished, Available)



```python

kustoOptions = {"kustoCluster": KUSTO_CLUSTER, "kustoDatabase" :KUSTO_DATABASE, "kustoTable" : KUSTO_TABLE }

# Replace the auth method based on your desired authentication mechanism  - https://github.com/Azure/azure-kusto-spark/blob/master/docs/Authentication.md
access_token=mssparkutils.credentials.getToken(kustoOptions["kustoCluster"])
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 21, Finished, Available)



```python
#Pandas data frame to spark dataframe
sparkDF=spark.createDataFrame(article_df)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 22, Finished, Available)


    /opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:604: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.



```python
# Write data to a Kusto table
sparkDF.write. \
format("com.microsoft.kusto.spark.synapse.datasource"). \
option("kustoCluster",kustoOptions["kustoCluster"]). \
option("kustoDatabase",kustoOptions["kustoDatabase"]). \
option("kustoTable", kustoOptions["kustoTable"]). \
option("accessToken", access_token). \
option("tableCreateOptions", "CreateIfNotExist").\
mode("Append"). \
save()

```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 23, Finished, Available)


### Prepare your OpenAI API key
# 

The OpenAI API key is used for vectorization of the documents and queries. You can follow the instructions to create and retrieve your Azure OpenAI key and endpoint. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/tutorials/embeddings


Please make sure to use the `text-embedding-3-small` model. Since the precomputed embeddings were created with `text-embedding-3-small` model we also have to use it during search.



```python
import openai
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 43, Finished, Available)


#### If using Azure Open AI


key: _node_content
string_value:
{"id_": "a9706389-0a15-472c-ba62-83958c2b4e72", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b92cdcf7-e9c6-4d59-8d89-0cf0d20e8b91", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f8b8e68a081daf61f29d4cfa647c6bfcafda5d048601a7c480a7958b3a28fb7e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66cfe7d7-e69d-477a-ace9-2e999eba090c", "node_type": "1", "metadata": {}, "hash": "c9e8e5be9bcf5fb8b27da96d341ecd7741ceaa63c9f0b91a92423093ebafaafc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13734, "end_char_idx": 16225, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 22, Finished, Available)


    /opt/spark/python/lib/pyspark.zip/pyspark/sql/pandas/conversion.py:604: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.



```python
# Write data to a Kusto table
sparkDF.write. \
format("com.microsoft.kusto.spark.synapse.datasource"). \
option("kustoCluster",kustoOptions["kustoCluster"]). \
option("kustoDatabase",kustoOptions["kustoDatabase"]). \
option("kustoTable", kustoOptions["kustoTable"]). \
option("accessToken", access_token). \
option("tableCreateOptions", "CreateIfNotExist").\
mode("Append"). \
save()

```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 23, Finished, Available)


### Prepare your OpenAI API key
# 

The OpenAI API key is used for vectorization of the documents and queries. You can follow the instructions to create and retrieve your Azure OpenAI key and endpoint. https://learn.microsoft.com/en-us/azure/cognitive-services/openai/tutorials/embeddings


Please make sure to use the `text-embedding-3-small` model. Since the precomputed embeddings were created with `text-embedding-3-small` model we also have to use it during search.



```python
import openai
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 43, Finished, Available)


#### If using Azure Open AI


```python
openai.api_version = '2022-12-01'
openai.api_base = '' # Please add your endpoint here
openai.api_type = 'azure'
openai.api_key = ''  # Please add your api key here

def embed(query):
    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
            input=query,
            deployment_id="embed", #replace with your deployment id
            chunk_size=1
    )["data"][0]["embedding"]
    return embedded_query
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 44, Finished, Available)


#### If using Open AI

Only run this cell if you plan to use Open AI for embedding


```python
openai.api_key = ""


def embed(query):
    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]
    return embedded_query
```

### Generate embedding for the search term


```python

searchedEmbedding = embed("places where you worship")
#print(searchedEmbedding)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 45, Finished, Available)


key: _node_content
string_value:
{"id_": "66cfe7d7-e69d-477a-ace9-2e999eba090c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9706389-0a15-472c-ba62-83958c2b4e72", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b497fe9b55707966419e162fa80311cfdce2b116a6bdfd96e31fe8cd81a0ad24", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6152b1b8-dd6a-4b85-b3da-7283f9392256", "node_type": "1", "metadata": {}, "hash": "f22cfe0ac1f9f23c98bf31f5fe975b784d2af3f489297996a9332ccd171d3f8c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14954, "end_char_idx": 17577, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import openai
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 43, Finished, Available)


#### If using Azure Open AI


```python
openai.api_version = '2022-12-01'
openai.api_base = '' # Please add your endpoint here
openai.api_type = 'azure'
openai.api_key = ''  # Please add your api key here

def embed(query):
    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
            input=query,
            deployment_id="embed", #replace with your deployment id
            chunk_size=1
    )["data"][0]["embedding"]
    return embedded_query
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 44, Finished, Available)


#### If using Open AI

Only run this cell if you plan to use Open AI for embedding


```python
openai.api_key = ""


def embed(query):
    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]
    return embedded_query
```

### Generate embedding for the search term


```python

searchedEmbedding = embed("places where you worship")
#print(searchedEmbedding)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 45, Finished, Available)


#### Semantic search in Kusto 

We will search the Kusto table for the closest vectors.

We will be using the series-cosine-similarity-fl UDF for similarity search. 

Please create the function in your database before proceeding -
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/functions-library/series-cosine-similarity-fl?tabs=query-defined


```python
from azure.kusto.data import KustoClient, KustoConnectionStringBuilder
from azure.kusto.data.exceptions import KustoServiceError
from azure.kusto.data.helpers import dataframe_from_result_table
import pandas as pd
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 35, Finished, Available)



```python
KCSB = KustoConnectionStringBuilder.with_aad_device_authentication(
    KUSTO_CLUSTER)
KCSB.authority_id = AAD_TENANT_ID
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 38, Finished, Available)



```python
KUSTO_CLIENT = KustoClient(KCSB)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 39, Finished, Available)



```python
KUSTO_QUERY = "Wiki | extend similarity = series_cosine_similarity_fl(dynamic("+str(searchedEmbedding)+"), content_vector,1,1) | top 10 by similarity desc "

RESPONSE = KUSTO_CLIENT.execute(KUSTO_DATABASE, KUSTO_QUERY)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 48, Finished, Available)


key: _node_content
string_value:
{"id_": "6152b1b8-dd6a-4b85-b3da-7283f9392256", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66cfe7d7-e69d-477a-ace9-2e999eba090c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "20263772450e0b8ec4c0923390aaded277dd5785a815b87f524c5c6eca7dac46", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2818d8a6-4cfa-488d-a2f9-73253090aa05", "node_type": "1", "metadata": {}, "hash": "f1b7445c92894888646041223e0b0ffcb3bb906c10bcbf4c255d6c4e61e0093c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16586, "end_char_idx": 17744, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from azure.kusto.data import KustoClient, KustoConnectionStringBuilder
from azure.kusto.data.exceptions import KustoServiceError
from azure.kusto.data.helpers import dataframe_from_result_table
import pandas as pd
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 35, Finished, Available)



```python
KCSB = KustoConnectionStringBuilder.with_aad_device_authentication(
    KUSTO_CLUSTER)
KCSB.authority_id = AAD_TENANT_ID
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 38, Finished, Available)



```python
KUSTO_CLIENT = KustoClient(KCSB)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 39, Finished, Available)



```python
KUSTO_QUERY = "Wiki | extend similarity = series_cosine_similarity_fl(dynamic("+str(searchedEmbedding)+"), content_vector,1,1) | top 10 by similarity desc "

RESPONSE = KUSTO_CLIENT.execute(KUSTO_DATABASE, KUSTO_QUERY)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 48, Finished, Available)



```python
df = dataframe_from_result_table(RESPONSE.primary_results[0])
df
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 49, Finished, Available)


key: _node_content
string_value:
{"id_": "2818d8a6-4cfa-488d-a2f9-73253090aa05", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6152b1b8-dd6a-4b85-b3da-7283f9392256", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2072e674a265591a1f1f1de58548e62dff5aa191ee06c689b1950a9146fb85b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3ac92bb-84a5-4d53-83a5-61d727d45f49", "node_type": "1", "metadata": {}, "hash": "fdde13f20a2be24c2f3d0e8064fbfbef7b1d4f4369356c3db5e3a7d40642c503", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16586, "end_char_idx": 19642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from azure.kusto.data import KustoClient, KustoConnectionStringBuilder
from azure.kusto.data.exceptions import KustoServiceError
from azure.kusto.data.helpers import dataframe_from_result_table
import pandas as pd
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 35, Finished, Available)



```python
KCSB = KustoConnectionStringBuilder.with_aad_device_authentication(
    KUSTO_CLUSTER)
KCSB.authority_id = AAD_TENANT_ID
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 38, Finished, Available)



```python
KUSTO_CLIENT = KustoClient(KCSB)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 39, Finished, Available)



```python
KUSTO_QUERY = "Wiki | extend similarity = series_cosine_similarity_fl(dynamic("+str(searchedEmbedding)+"), content_vector,1,1) | top 10 by similarity desc "

RESPONSE = KUSTO_CLIENT.execute(KUSTO_DATABASE, KUSTO_QUERY)
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 48, Finished, Available)



```python
df = dataframe_from_result_table(RESPONSE.primary_results[0])
df
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 49, Finished, Available)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
      <th>similarity</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>852</td>
      <td>https://simple.wikipedia.org/wiki/Temple</td>
      <td>Temple</td>
      <td>A temple is a building where people go to prac...</td>
      <td>[-0.021837441250681877, -0.007722342386841774,...</td>
      <td>[-0.0019541378132998943, 0.007151313126087189,...</td>
      <td>413</td>
      <td>0.834495</td>
    </tr>
    <tr>
      <th>1</th>
      <td>78094</td>
      <td>https://simple.wikipedia.org/wiki/Christian%20...</td>
      <td>Christian worship</td>
      <td>In Christianity, worship has been thought as b...</td>
      <td>[0.0017675267299637198, -0.008890199474990368,...</td>
      <td>[0.020530683919787407, 0.0024345638230443, -0....</td>
      <td>20320</td>
      <td>0.832132</td>
    </tr>
    <tr>
      <th>2</th>
      <td>59154</td>
      <td>https://simple.wikipedia.org/wiki/Service%20of...</td>
      <td>Service of worship</td>
      <td>A service of worship is a religious meeting wh...</td>
      <td>[-0.007969820871949196, 0.0004240311391185969,...</td>
      <td>[0.003784010885283351, -0.0030924836173653603,...</td>
      <td>15519</td>
      <td>0.831633</td>
    </tr>
    <tr>
      <th>3</th>
      <td>51910</td>
      <td>https://simple.wikipedia.org/wiki/Worship</td>
      <td>Worship</td>
      <td>Worship is a word often used in religion.


key: _node_content
string_value:
{"id_": "e3ac92bb-84a5-4d53-83a5-61d727d45f49", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2818d8a6-4cfa-488d-a2f9-73253090aa05", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b10fa5e3a4e609b43e476f5efe4f4f90e640694a02330400c2c8a1ac14bbf94d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97bb54c9-2446-4ab6-b3cc-da7caba70bb2", "node_type": "1", "metadata": {}, "hash": "4049fe6e800733c814f0f34a9039f72c9bd1b4f4dd37eeeeec45e0e6afb661fb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19644, "end_char_idx": 21533, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It ...</td>
      <td>[0.0036036288365721703, -0.01276545226573944, ...</td>
      <td>[0.007925753481686115, -0.0110504487529397, 0....</td>
      <td>14010</td>
      <td>0.828185</td>
    </tr>
    <tr>
      <th>4</th>
      <td>29576</td>
      <td>https://simple.wikipedia.org/wiki/Altar</td>
      <td>Altar</td>
      <td>An altar is a place, often a table, where a re...</td>
      <td>[0.007887467741966248, -0.02706138789653778, -...</td>
      <td>[0.023901859298348427, -0.031175222247838977, ...</td>
      <td>8708</td>
      <td>0.824124</td>
    </tr>
    <tr>
      <th>5</th>
      <td>92507</td>
      <td>https://simple.wikipedia.org/wiki/Shrine</td>
      <td>Shrine</td>
      <td>A shrine is a holy or sacred place with someth...</td>
      <td>[-0.011601685546338558, 0.006366696208715439, ...</td>
      <td>[0.016423320397734642, -0.0015560361789539456,...</td>
      <td>23945</td>
      <td>0.823863</td>
    </tr>
    <tr>
      <th>6</th>
      <td>815</td>
      <td>https://simple.wikipedia.org/wiki/Synagogue</td>
      <td>Synagogue</td>
      <td>A synagogue is a place where Jews meet to wors...</td>
      <td>[-0.017317570745944977, 0.0022673190105706453,...</td>
      <td>[-0.004515442531555891, 0.003739549545571208, ...</td>
      <td>398</td>
      <td>0.819942</td>
    </tr>
    <tr>
      <th>7</th>
      <td>68080</td>
      <td>https://simple.wikipedia.org/wiki/Shinto%20shrine</td>
      <td>Shinto shrine</td>
      <td>A Shinto shrine is a sacred place or site wher...</td>
      <td>[0.0035740730818361044, 0.0028098472394049168,...</td>
      <td>[0.011014971882104874, 0.00042272370774298906,...</td>
      <td>18106</td>
      <td>0.818475</td>
    </tr>
    <tr>
      <th>8</th>
      <td>57790</td>
      <td>https://simple.wikipedia.org/wiki/Chapel</td>
      <td>Chapel</td>
      <td>A chapel is a place for Christian worship.


key: _node_content
string_value:
{"id_": "97bb54c9-2446-4ab6-b3cc-da7caba70bb2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3ac92bb-84a5-4d53-83a5-61d727d45f49", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2a46943041f73d822cc4c540eebbdf2295096ef251d05c14a50b529d64eab307", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e92cccf-34c0-45bd-b50d-f0e663839b12", "node_type": "1", "metadata": {}, "hash": "e0933c6dfc9392ec545bb692e8b53e96ff82a5d58ee7b62053e6f39937bf838b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The...</td>
      <td>[-0.01371884811669588, 0.0031672674231231213, ...</td>
      <td>[0.002526090247556567, 0.02482965588569641, 0....</td>
      <td>15260</td>
      <td>0.817608</td>
    </tr>
    <tr>
      <th>9</th>
      <td>142</td>
      <td>https://simple.wikipedia.org/wiki/Church%20%28...</td>
      <td>Church (building)</td>
      <td>A church is a building that was constructed to...</td>
      <td>[0.0021336888894438744, 0.0029748091474175453,...</td>
      <td>[0.016109377145767212, 0.022908871993422508, 0...</td>
      <td>74</td>
      <td>0.812636</td>
    </tr>
  </tbody>
</table>
</div>




```python
searchedEmbedding = embed("unfortunate events in history")

```


```python
KUSTO_QUERY = "Wiki | extend similarity = series_cosine_similarity_fl(dynamic("+str(searchedEmbedding)+"), title_vector,1,1) | top 10 by similarity desc "
RESPONSE = KUSTO_CLIENT.execute(KUSTO_DATABASE, KUSTO_QUERY)

df = dataframe_from_result_table(RESPONSE.primary_results[0])
df
```


    StatementMeta(, 7e5070d2-4560-4fb8-a3a8-6a594acd58ab, 52, Finished, Available)





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
      <th>similarity</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>848</td>
      <td>https://simple.wikipedia.org/wiki/Tragedy</td>
      <td>Tragedy</td>
      <td>In theatre, a tragedy as defined by Aristotle .</td>
      <td>[-0.019502468407154083, -0.010160734876990318,</td>
      <td>[-0.012951433658599854, -0.018836138769984245,</td>
      <td>410</td>
      <td>0.851848</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4469</td>
      <td>https://simple.wikipedia.org/wiki/The%20Holocaust</td>
      <td>The Holocaust</td>
      <td>The Holocaust, sometimes called The Shoah (), .


key: _node_content
string_value:
{"id_": "0e92cccf-34c0-45bd-b50d-f0e663839b12", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97bb54c9-2446-4ab6-b3cc-da7caba70bb2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c1c12aa1b6c612449a5fb51305e0e5892224d157471edd23ca34010a9c87c482", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "251d9e2d-f4d2-4157-b839-1d1fb9789a84", "node_type": "1", "metadata": {}, "hash": "417340c4907f781f8894eb80f27dc594cb70dddf5021d72057f71716d3cb1c21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
      <th>similarity</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>848</td>
      <td>https://simple.wikipedia.org/wiki/Tragedy</td>
      <td>Tragedy</td>
      <td>In theatre, a tragedy as defined by Aristotle .</td>
      <td>[-0.019502468407154083, -0.010160734876990318,</td>
      <td>[-0.012951433658599854, -0.018836138769984245,</td>
      <td>410</td>
      <td>0.851848</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4469</td>
      <td>https://simple.wikipedia.org/wiki/The%20Holocaust</td>
      <td>The Holocaust</td>
      <td>The Holocaust, sometimes called The Shoah (), .</td>
      <td>[-0.030233195051550865, -0.024401605129241943,</td>
      <td>[-0.016398731619119644, -0.013267949223518372,</td>
      <td>1203</td>
      <td>0.847222</td>
    </tr>
    <tr>
      <th>2</th>
      <td>64216</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of historical plagues</td>
      <td>This list contains famous or well documented o.</td>
      <td>[-0.010667890310287476, -0.0003575817099772393.</td>
      <td>[-0.010863155126571655, -0.0012196656316518784.</td>
      <td>16859</td>
      <td>0.844411</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4397</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of disasters</td>
      <td>This is a list of disasters, both natural and .</td>
      <td>[-0.02713736332952976, -0.005278210621327162, .</td>
      <td>[-0.023679986596107483, -0.006126823835074902,</td>
      <td>1158</td>
      <td>0.


key: _node_content
string_value:
{"id_": "251d9e2d-f4d2-4157-b839-1d1fb9789a84", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e92cccf-34c0-45bd-b50d-f0e663839b12", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "76fb7eca4d494907c24da3dd511ee504d09123ce39a5976e0adcbc0f659ee473", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a11bb1e-a42f-482c-8a38-61d8611c3bfb", "node_type": "1", "metadata": {}, "hash": "77c1b8171812972be76d3182c6486c2f84d822a4791d3aa429aa7714cd3bf9c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
sometimes called The Shoah (), .</td>
      <td>[-0.030233195051550865, -0.024401605129241943,</td>
      <td>[-0.016398731619119644, -0.013267949223518372,</td>
      <td>1203</td>
      <td>0.847222</td>
    </tr>
    <tr>
      <th>2</th>
      <td>64216</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of historical plagues</td>
      <td>This list contains famous or well documented o.</td>
      <td>[-0.010667890310287476, -0.0003575817099772393.</td>
      <td>[-0.010863155126571655, -0.0012196656316518784.</td>
      <td>16859</td>
      <td>0.844411</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4397</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of disasters</td>
      <td>This is a list of disasters, both natural and .</td>
      <td>[-0.02713736332952976, -0.005278210621327162, .</td>
      <td>[-0.023679986596107483, -0.006126823835074902,</td>
      <td>1158</td>
      <td>0.843063</td>
    </tr>
    <tr>
      <th>4</th>
      <td>23073</td>
      <td>https://simple.wikipedia.org/wiki/Disaster</td>
      <td>Disaster</td>
      <td>A disaster is something very not good that hap.</td>
      <td>[-0.018235962837934497, -0.020034968852996823,</td>
      <td>[-0.02504003793001175, 0.007415903266519308, 0.</td>
      <td>7251</td>
      <td>0.840334</td>
    </tr>
    <tr>
      <th>5</th>
      <td>4382</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of terrorist incidents</td>
      <td>The following is a list by date of acts and fa.</td>
      <td>[-0.03989032283425331, -0.012808636762201786, .</td>
      <td>[-0.045838188380002975, -0.01682935282588005, .</td>
      <td>1149</td>
      <td>0.836162</td>
    </tr>
    <tr>
      <th>6</th>
      <td>13528</td>
      <td>https://simple.wikipedia.org/wiki/A%20Series%2.</td>
      <td>A Series of Unfortunate Events</td>
      <td>A Series of Unfortunate Events is a series of .</td>
      <td>[0.


key: _node_content
string_value:
{"id_": "3a11bb1e-a42f-482c-8a38-61d8611c3bfb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "251d9e2d-f4d2-4157-b839-1d1fb9789a84", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e3dde47d5b4a92094ffe3765f5a5fa217029da304ac2db9f7ca0585999eb03f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f86b614-e286-4ef9-b083-8f5612463cb3", "node_type": "1", "metadata": {}, "hash": "21d1b12e6b36bf914c92e15aa2b66237ef51bd2a79c391324eab94432268a216", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
843063</td>
    </tr>
    <tr>
      <th>4</th>
      <td>23073</td>
      <td>https://simple.wikipedia.org/wiki/Disaster</td>
      <td>Disaster</td>
      <td>A disaster is something very not good that hap.</td>
      <td>[-0.018235962837934497, -0.020034968852996823,</td>
      <td>[-0.02504003793001175, 0.007415903266519308, 0.</td>
      <td>7251</td>
      <td>0.840334</td>
    </tr>
    <tr>
      <th>5</th>
      <td>4382</td>
      <td>https://simple.wikipedia.org/wiki/List%20of%20.</td>
      <td>List of terrorist incidents</td>
      <td>The following is a list by date of acts and fa.</td>
      <td>[-0.03989032283425331, -0.012808636762201786, .</td>
      <td>[-0.045838188380002975, -0.01682935282588005, .</td>
      <td>1149</td>
      <td>0.836162</td>
    </tr>
    <tr>
      <th>6</th>
      <td>13528</td>
      <td>https://simple.wikipedia.org/wiki/A%20Series%2.</td>
      <td>A Series of Unfortunate Events</td>
      <td>A Series of Unfortunate Events is a series of .</td>
      <td>[0.0010618815431371331, -0.0267023965716362, -.</td>
      <td>[0.002801976166665554, -0.02904471382498741, -.</td>
      <td>4347</td>
      <td>0.835172</td>
    </tr>
    <tr>
      <th>7</th>
      <td>42874</td>
      <td>https://simple.wikipedia.org/wiki/History%20of.</td>
      <td>History of the world</td>
      <td>The history of the world (also called human hi.</td>
      <td>[0.0026915925554931164, -0.022206028923392296,</td>
      <td>[0.013645033352077007, -0.005165994167327881, .</td>
      <td>11672</td>
      <td>0.830243</td>
    </tr>
    <tr>
      <th>8</th>
      <td>4452</td>
      <td>https://simple.wikipedia.org/wiki/Accident</td>
      <td>Accident</td>
      <td>An accident is when something goes wrong when .</td>
      <td>[-0.004075294826179743, -0.0059883203357458115.</td>
      <td>[0.00926120299845934, 0.013705797493457794, 0.</td>
      <td>1190</td>
      <td>0.


key: _node_content
string_value:
{"id_": "3f86b614-e286-4ef9-b083-8f5612463cb3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7ebfd80-d81a-43ac-aeff-0bf8c7a35eba", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "44338ce11295cc314f900d37b467b05e05e490a8bac9fe71e0eacb53ae8831fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a11bb1e-a42f-482c-8a38-61d8611c3bfb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/Getting_started_with_kusto_and_openai_embeddings.ipynb", "title": "Kusto as a Vector database for AI embeddings", "tokens": 8919, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9b71e437ceda866bd3bb59110b799d70ddd303474971f3520360ea5eec4d5419", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>A Series of Unfortunate Events</td>
      <td>A Series of Unfortunate Events is a series of .</td>
      <td>[0.0010618815431371331, -0.0267023965716362, -.</td>
      <td>[0.002801976166665554, -0.02904471382498741, -.</td>
      <td>4347</td>
      <td>0.835172</td>
    </tr>
    <tr>
      <th>7</th>
      <td>42874</td>
      <td>https://simple.wikipedia.org/wiki/History%20of.</td>
      <td>History of the world</td>
      <td>The history of the world (also called human hi.</td>
      <td>[0.0026915925554931164, -0.022206028923392296,</td>
      <td>[0.013645033352077007, -0.005165994167327881, .</td>
      <td>11672</td>
      <td>0.830243</td>
    </tr>
    <tr>
      <th>8</th>
      <td>4452</td>
      <td>https://simple.wikipedia.org/wiki/Accident</td>
      <td>Accident</td>
      <td>An accident is when something goes wrong when .</td>
      <td>[-0.004075294826179743, -0.0059883203357458115.</td>
      <td>[0.00926120299845934, 0.013705797493457794, 0.</td>
      <td>1190</td>
      <td>0.826898</td>
    </tr>
    <tr>
      <th>9</th>
      <td>324</td>
      <td>https://simple.wikipedia.org/wiki/History</td>
      <td>History</td>
      <td>History is the study of past events. People kn...</td>
      <td>[0.006603690329939127, -0.011856242083013058, ...</td>
      <td>[0.0048830462619662285, 0.0032003086525946856,...</td>
      <td>170</td>
      <td>0.824645</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "e1d219a4-5a3e-4464-8787-16d2c6b2c2cc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/README.ipynb", "title": "Kusto as a Vector database", "tokens": 363, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1a429034-77b6-4e73-8a76-d1d6a8931886", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/kusto/README.ipynb", "title": "Kusto as a Vector database", "tokens": 363, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a2b8cbb1ef95c86087b64b18cbf9cccfce14bb3ebc9bcc9a72bfa48db6a0d6e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1681, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Kusto as a Vector database



[Azure Data Explorer aka Kusto](https://azure.microsoft.com/en-us/products/data-explorer) is a cloud-based data analytics service that enables users to perform advanced analytics on large datasets in real-time. It is particularly well-suited for handling large volumes of data, making it an excellent choice for storing and searching vectors.

Kusto supports a special data type called dynamic, which can store unstructured data such as arrays and properties bag. [Dynamic data type](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/dynamic) is perfect for storing vector values. You can further augment the vector value by storing metadata related to the original object as separate columns in your table.  
Kusto also supports in-built function [series_cosine_similarity_fl](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/functions-library/series-cosine-similarity-fl) to perform vector similarity searches.

[Get started](https://aka.ms/kustofree) with Kusto for free. 

![Kusto_Vector](./images/kusto_vector_db.png)



## Getting started with Kusto and Open AI embedding

### Demo Scenario

![Wiki_embeddings](./images/wiki_embeddings.png)

![semantic_search_flow](./images/semantic_search_user_flow.png)

If you’d like to try this demo, please follow the instructions in the [Notebook](Getting_started_with_kusto_and_openai_embeddings.ipynb).

It will allow you to -  

1. Use precomputed embeddings created by OpenAI API. 

2. Store the embeddings in Kusto. 

3. Convert raw text query to an embedding with OpenAI API. 

4. Use Kusto to perform cosine similarity search in the stored embeddings.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "44e2260b-287a-4316-ae76-8ea115ab1eb3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77d831a4-cfa6-4fba-ab00-06cc49371af7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2ece695e32dc0c859481d21bcdb2f8f0c59c61e95c8d611f90bcaf52ac574f93", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a212d312-a5db-4d75-9492-37ddf7c3a862", "node_type": "1", "metadata": {}, "hash": "2236303d6b8edacf79adf5cde2a465109b1bc6566bf95b6a89f9bc36006f737f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3307, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Hologres as a vector database for OpenAI embeddings

This notebook guides you step by step on using Hologres as a vector database for OpenAI embeddings.

This notebook presents an end-to-end process of:
1. Using precomputed embeddings created by OpenAI API.
2. Storing the embeddings in a cloud instance of Hologres.
3. Converting raw text query to an embedding with OpenAI API.
4. Using Hologres to perform the nearest neighbour search in the created collection.
5. Provide large language models with the search results as context in prompt engineering

### What is Hologres

[Hologres](https://www.alibabacloud.com/help/en/hologres/latest/what-is-hologres) is a unified real-time data warehousing service developed by Alibaba Cloud. You can use Hologres to write, update, process, and analyze large amounts of data in real time. Hologres supports standard SQL syntax, is compatible with PostgreSQL, and supports most PostgreSQL functions. Hologres supports online analytical processing (OLAP) and ad hoc analysis for up to petabytes of data, and provides high-concurrency and low-latency online data services. Hologres supports fine-grained isolation of multiple workloads and enterprise-level security capabilities. Hologres is deeply integrated with MaxCompute, Realtime Compute for Apache Flink, and DataWorks, and provides full-stack online and offline data warehousing solutions for enterprises.

Hologres provides vector database functionality by adopting [Proxima](https://www.alibabacloud.com/help/en/hologres/latest/vector-processing).

Proxima is a high-performance software library developed by Alibaba DAMO Academy. It allows you to search for the nearest neighbors of vectors. Proxima provides higher stability and performance than similar open source software such as Facebook AI Similarity Search (Faiss). Proxima provides basic modules that have leading performance and effects in the industry and allows you to search for similar images, videos, or human faces. Hologres is deeply integrated with Proxima to provide a high-performance vector search service.

### Deployment options

- [Click here](https://www.alibabacloud.com/product/hologres) to fast deploy [Hologres data warehouse](https://www.alibabacloud.com/help/en/hologres/latest/getting-started).


## Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. Hologres cloud server instance.
2. The 'psycopg2-binary' library to interact with the vector database. Any other postgresql client library is ok.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys).



We might validate if the server was launched successfully by running a simple curl command:


### Install requirements

This notebook obviously requires the `openai` and `psycopg2-binary` packages, but there are also some other additional libraries we will use. The following command installs them all:



```python
! pip install openai psycopg2-binary pandas wget
```

### Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


key: _node_content
string_value:
{"id_": "a212d312-a5db-4d75-9492-37ddf7c3a862", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77d831a4-cfa6-4fba-ab00-06cc49371af7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2ece695e32dc0c859481d21bcdb2f8f0c59c61e95c8d611f90bcaf52ac574f93", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44e2260b-287a-4316-ae76-8ea115ab1eb3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "34cd1b7e42c7ce6c83888cfa436ffaa7db32ea2b2c7d4420753d5ef6cacda0cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "111a23e9-c0f9-4bd1-887b-9ce5e9496caa", "node_type": "1", "metadata": {}, "hash": "c4105f6c713872c0885ec57f4cd51584f75041811d3e2871f13eb9ba7cfc715e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2286, "end_char_idx": 5357, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. Hologres cloud server instance.
2. The 'psycopg2-binary' library to interact with the vector database. Any other postgresql client library is ok.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys).



We might validate if the server was launched successfully by running a simple curl command:


### Install requirements

This notebook obviously requires the `openai` and `psycopg2-binary` packages, but there are also some other additional libraries we will use. The following command installs them all:



```python
! pip install openai psycopg2-binary pandas wget
```

### Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Connect to Hologres
First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below

Connecting to a running instance of Hologres server is easy with the official Python library:


```python
import os
import psycopg2

# Note. alternatively you can set a temporary env variable like this:
# os.environ["PGHOST"] = "your_host"
# os.environ["PGPORT"] "5432"),
# os.environ["PGDATABASE"] "postgres"),
# os.environ["PGUSER"] "user"),
# os.environ["PGPASSWORD"] "password"),

connection = psycopg2.connect(
    host=os.environ.get("PGHOST", "localhost"),
    port=os.environ.get("PGPORT", "5432"),
    database=os.environ.get("PGDATABASE", "postgres"),
    user=os.environ.get("PGUSER", "user"),
    password=os.environ.get("PGPASSWORD", "password")
)
connection.set_session(autocommit=True)

# Create a new cursor object
cursor = connection.cursor()
```

We can test the connection by running any available method:


```python

# Execute a simple query to test the connection
cursor.execute("SELECT 1;")
result = cursor.fetchone()

# Check the query result
if result == (1,):
    print("Connection successful!")
else:
    print("Connection failed.")
```

    Connection successful!



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

The downloaded file has to be then extracted:


key: _node_content
string_value:
{"id_": "111a23e9-c0f9-4bd1-887b-9ce5e9496caa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "77d831a4-cfa6-4fba-ab00-06cc49371af7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2ece695e32dc0c859481d21bcdb2f8f0c59c61e95c8d611f90bcaf52ac574f93", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a212d312-a5db-4d75-9492-37ddf7c3a862", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "55a76d9fba20b7b44f2517c4390623c8838af70de40c7b65a2a48ef8724187c4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be80db80-5f7e-46d4-8039-227caa4e667e", "node_type": "1", "metadata": {}, "hash": "66d9df8f734b22fb4b9d9e2088f6f8c0736f964dd981da6a018711f2c379a2ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3869, "end_char_idx": 6823, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Connect to Hologres
First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below

Connecting to a running instance of Hologres server is easy with the official Python library:


```python
import os
import psycopg2

# Note. alternatively you can set a temporary env variable like this:
# os.environ["PGHOST"] = "your_host"
# os.environ["PGPORT"] "5432"),
# os.environ["PGDATABASE"] "postgres"),
# os.environ["PGUSER"] "user"),
# os.environ["PGPASSWORD"] "password"),

connection = psycopg2.connect(
    host=os.environ.get("PGHOST", "localhost"),
    port=os.environ.get("PGPORT", "5432"),
    database=os.environ.get("PGDATABASE", "postgres"),
    user=os.environ.get("PGUSER", "user"),
    password=os.environ.get("PGPASSWORD", "password")
)
connection.set_session(autocommit=True)

# Create a new cursor object
cursor = connection.cursor()
```

We can test the connection by running any available method:


```python

# Execute a simple query to test the connection
cursor.execute("SELECT 1;")
result = cursor.fetchone()

# Check the query result
if result == (1,):
    print("Connection successful!")
else:
    print("Connection failed.")
```

    Connection successful!



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

The downloaded file has to be then extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")

```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


```python
!unzip -n vector_database_wikipedia_articles_embedded.zip
!ls -lh vector_database_wikipedia_articles_embedded.csv
```

    Archive:  vector_database_wikipedia_articles_embedded.zip
    -rw-r--r--@ 1 geng  staff   1.7G Jan 31 01:19 vector_database_wikipedia_articles_embedded.csv


Take a look at the data.


```python
import pandas, json
data = pandas.read_csv('../../data/vector_database_wikipedia_articles_embedded.csv')
data
```


