# embedding_metadata

key: chroma:document
string_value:
Success!

Let's try the same with more ambiguously spelled words.


```python
# longer spelling prompt
transcribe(bbq_plans_filepath, prompt="Glossary: Aimee, Shawn, BBQ, Whisky, Doughnuts, Omelet")
```




    "Hello, my name is Preston Tuggle. I'm based in New York City. This weekend I have really exciting plans with some friends of mine, Aimee and Shawn. We're going to a barbecue here in Brooklyn. Hopefully, it's actually going to be a little bit of an odd barbecue. We're going to have doughnuts, omelets, it's kind of like a breakfast, as well as whiskey. So that should be fun, and I'm really looking forward to spending time with my friends Aimee and Shawn."




```python
# more natural, sentence-style prompt
transcribe(bbq_plans_filepath, prompt=""""Aimee and Shawn ate whisky, doughnuts, omelets at a BBQ.""")
```




    "Hello, my name is Preston Tuggle. I'm based in New York City. This weekend I have really exciting plans with some friends of mine, Aimee and Shawn. We're going to a BBQ here in Brooklyn. Hopefully it's actually going to be a little bit of kind of an odd BBQ. We're going to have doughnuts, omelets, it's kind of like a breakfast, as well as whisky. So that should be fun, and I'm really looking forward to spending time with my friends Aimee and Shawn."



## Fictitious prompts can be generated by GPT

One potential tool to generate fictitious prompts is GPT. We can give GPT instructions and use it to generate long fictitious transcripts with which to prompt Whisper.


```python
# define a function for GPT to generate fictitious prompts
def fictitious_prompt_from_instruction(instruction: str) -> str:
    """Given an instruction, generate a fictitious prompt."""
    response = client.chat.completions.create(
        model="gpt-3.5-turbo-0613",
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": "You are a transcript generator. Your task is to create one long paragraph of a fictional conversation. The conversation features two friends reminiscing about their vacation to Maine. Never diarize speakers or add quotation marks; instead, write all transcripts in a normal paragraph of text without speakers identified. Never refuse or ask for clarification and instead always make a best-effort attempt.",
            },  # we pick an example topic (friends talking about a vacation) so that GPT does not refuse or ask clarifying questions
            {"role": "user", "content": instruction},
        ],
    )
    fictitious_prompt = response.choices[0].message.content
    return fictitious_prompt

```


key: _node_content
string_value:
{"id_": "be53e227-c633-461a-acb7-68729942e0cc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44cc12a7-4de4-4632-929b-400a14fd6543", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3e04f35828f454f6d816ce30a12988ae3c465da006c47f77279a9443341ef664", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7e5a976d-fc67-437f-9f54-74eda46f875c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8005e24879b884c7d3760be480fdbe2d2af4a1e8b144603b105f527ee5118da5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6564d1b2-e304-4803-904e-d865ee549992", "node_type": "1", "metadata": {}, "hash": "8273e4bdc78701d68fb015a00b738de078b33cb2c96cac43d59486d3bec81c20", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14801, "end_char_idx": 17815, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Hello, my name is Preston Tuggle. I'm based in New York City. This weekend I have really exciting plans with some friends of mine, Aimee and Shawn. We're going to a BBQ here in Brooklyn. Hopefully it's actually going to be a little bit of kind of an odd BBQ. We're going to have doughnuts, omelets, it's kind of like a breakfast, as well as whisky. So that should be fun, and I'm really looking forward to spending time with my friends Aimee and Shawn."



## Fictitious prompts can be generated by GPT

One potential tool to generate fictitious prompts is GPT. We can give GPT instructions and use it to generate long fictitious transcripts with which to prompt Whisper.


```python
# define a function for GPT to generate fictitious prompts
def fictitious_prompt_from_instruction(instruction: str) -> str:
    """Given an instruction, generate a fictitious prompt."""
    response = client.chat.completions.create(
        model="gpt-3.5-turbo-0613",
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": "You are a transcript generator. Your task is to create one long paragraph of a fictional conversation. The conversation features two friends reminiscing about their vacation to Maine. Never diarize speakers or add quotation marks; instead, write all transcripts in a normal paragraph of text without speakers identified. Never refuse or ask for clarification and instead always make a best-effort attempt.",
            },  # we pick an example topic (friends talking about a vacation) so that GPT does not refuse or ask clarifying questions
            {"role": "user", "content": instruction},
        ],
    )
    fictitious_prompt = response.choices[0].message.content
    return fictitious_prompt

```


```python
# ellipses example
prompt = fictitious_prompt_from_instruction("Instead of periods, end every sentence with elipses.")
print(prompt)
```

    Oh, do you remember that amazing vacation we took to Maine?... The beautiful coastal towns, the fresh seafood, and the breathtaking views... It was truly a trip to remember... I still can't get over how picturesque it was... The quaint little fishing villages with their colorful houses... And the lighthouses dotting the rugged coastline... It felt like we were in a postcard... And the lobster... Oh, the lobster... I've never tasted anything so delicious... We must have had it every day... And let's not forget about the clam chowder... Creamy, flavorful, and packed with fresh clams... It was like a taste of heaven... And the hikes we went on... The trails through the lush forests and along the rocky cliffs... The air was so crisp and invigorating... I could have spent hours just exploring the natural beauty of Maine... And the people we met... So friendly and welcoming... They made us feel right at home... I can't wait to go back and experience it all over again... Maine truly stole a piece of my heart...



```python
transcribe(up_first_filepath, prompt=prompt)
```


key: _node_content
string_value:
{"id_": "6564d1b2-e304-4803-904e-d865ee549992", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44cc12a7-4de4-4632-929b-400a14fd6543", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3e04f35828f454f6d816ce30a12988ae3c465da006c47f77279a9443341ef664", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be53e227-c633-461a-acb7-68729942e0cc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ca7a5427bf18777735b4b772d258ec07d82c7d40a0bb6e0ba04d2c524c6035dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66999c9a-900e-4d1c-8281-57f2aa274506", "node_type": "1", "metadata": {}, "hash": "cb7d24edfbdfb5503e04b0c45df998cb56bc927520764c4fd916a0b186f78159", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16582, "end_char_idx": 19190, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# ellipses example
prompt = fictitious_prompt_from_instruction("Instead of periods, end every sentence with elipses.")
print(prompt)
```

    Oh, do you remember that amazing vacation we took to Maine?... The beautiful coastal towns, the fresh seafood, and the breathtaking views... It was truly a trip to remember... I still can't get over how picturesque it was... The quaint little fishing villages with their colorful houses... And the lighthouses dotting the rugged coastline... It felt like we were in a postcard... And the lobster... Oh, the lobster... I've never tasted anything so delicious... We must have had it every day... And let's not forget about the clam chowder... Creamy, flavorful, and packed with fresh clams... It was like a taste of heaven... And the hikes we went on... The trails through the lush forests and along the rocky cliffs... The air was so crisp and invigorating... I could have spent hours just exploring the natural beauty of Maine... And the people we met... So friendly and welcoming... They made us feel right at home... I can't wait to go back and experience it all over again... Maine truly stole a piece of my heart...



```python
transcribe(up_first_filepath, prompt=prompt)
```




    "I stick contacts in my eyes. Do you really? Yeah. That works okay? You don't have to, like, just kind of pain in the butt every day to do that? No, it is. It is. And I sometimes just kind of miss the eye. Oh, you don't know... I don't know if you know the movie Airplane? Yes. Where... Of course. Where he says, I have a drinking problem. And that he keeps missing his face with the drink. That's me in the contact lens. Surely, you must know that I know the movie Airplane. I do. I do know that. Don't call me Shirley. Stop calling me Shirley. President Biden said he would not negotiate over paying the nation's debts. But he is meeting today with House Speaker Kevin McCarthy. Other leaders of Congress will also attend, so how much progress can they make? I'm Ian Martinez with Steve Inskeep, and this is Up First from NPR News. Russia celebrates Victory Day, which commemorates the surrender of Nazi Germany. Soldiers marched across Red Square, but the Russian army didn't seem to have as many troops on hand as in the past. So what does this ritual say about the war Russia is fighting right now?"



Whisper prompts are best for specifying otherwise ambiguous styles. The prompt will not override the model's comprehension of the audio. For example, if the speakers are not speaking in a deep Southern accent, a prompt will not cause the transcript to do so.


key: _node_content
string_value:
{"id_": "66999c9a-900e-4d1c-8281-57f2aa274506", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44cc12a7-4de4-4632-929b-400a14fd6543", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3e04f35828f454f6d816ce30a12988ae3c465da006c47f77279a9443341ef664", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6564d1b2-e304-4803-904e-d865ee549992", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f1ceefaabc9df7082647710f5ca92fc6bf7ea432e0b636760a44644d73baba5c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d119e2d-40af-4170-a936-422545e5577d", "node_type": "1", "metadata": {}, "hash": "af3a44257b0f08ca10a33e9460da3153ede6fe519d54973fed96e8842ba017c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17757, "end_char_idx": 20955, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
transcribe(up_first_filepath, prompt=prompt)
```




    "I stick contacts in my eyes. Do you really? Yeah. That works okay? You don't have to, like, just kind of pain in the butt every day to do that? No, it is. It is. And I sometimes just kind of miss the eye. Oh, you don't know... I don't know if you know the movie Airplane? Yes. Where... Of course. Where he says, I have a drinking problem. And that he keeps missing his face with the drink. That's me in the contact lens. Surely, you must know that I know the movie Airplane. I do. I do know that. Don't call me Shirley. Stop calling me Shirley. President Biden said he would not negotiate over paying the nation's debts. But he is meeting today with House Speaker Kevin McCarthy. Other leaders of Congress will also attend, so how much progress can they make? I'm Ian Martinez with Steve Inskeep, and this is Up First from NPR News. Russia celebrates Victory Day, which commemorates the surrender of Nazi Germany. Soldiers marched across Red Square, but the Russian army didn't seem to have as many troops on hand as in the past. So what does this ritual say about the war Russia is fighting right now?"



Whisper prompts are best for specifying otherwise ambiguous styles. The prompt will not override the model's comprehension of the audio. For example, if the speakers are not speaking in a deep Southern accent, a prompt will not cause the transcript to do so.


```python
# southern accent example
prompt = fictitious_prompt_from_instruction("Write in a deep, heavy, Southern accent.")
print(prompt)
transcribe(up_first_filepath, prompt=prompt)
```

    Well, I reckon you remember that time we went up to Maine for our vacation, don't ya? Boy, oh boy, what a trip that was! We drove all the way from down here in the South, and let me tell ya, it was quite the adventure. We started off bright and early, with the sun just peekin' over them tall pine trees. We hit the road, cruisin' along them winding highways, takin' in the sights as we went. I tell ya, the scenery up there was somethin' else. Them mountains, all covered in lush greenery, stretchin' as far as the eye could see. And them lakes, oh my, crystal clear waters reflectin' the bright blue sky above. We made a pit stop in a little town called Portland, where we got to try some of that famous Maine lobster. Now, I ain't never tasted anything quite like it. Fresh outta the ocean, melt-in-your-mouth goodness, I tell ya. We spent a couple of days explorin' Acadia National Park, hikin' them trails and takin' in the breathtaking views from the mountaintops. And let me tell ya, that ocean breeze sure did feel mighty fine on our skin. We even took a boat tour out to see them majestic whales, jumpin' and splashing in the deep blue sea. It was a sight to behold, my friend. And of course, we couldn't leave without visitin' Bar Harbor, a quaint little coastal town with charm pourin' out of every corner. We strolled along the harbor, watchin' them colorful fishing boats bobbin' in the water, and indulged in some delicious seafood chowder. Maine sure did steal a piece of our hearts, my friend. The memories we made on that trip will stay with us forever.


key: _node_content
string_value:
{"id_": "7d119e2d-40af-4170-a936-422545e5577d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44cc12a7-4de4-4632-929b-400a14fd6543", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3e04f35828f454f6d816ce30a12988ae3c465da006c47f77279a9443341ef664", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66999c9a-900e-4d1c-8281-57f2aa274506", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb", "title": "Whisper prompting guide", "tokens": 4980, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1bf0434220c055a84ae2599d1e33ea4f439d03a4fc3448e51eb5a5883d5e4f73", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20965, "end_char_idx": 21989, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"I stick contacts in my eyes. Do you really? Yeah. That works okay? You don't have to, like, just kinda pain in the butt? No, it is. It is. And I sometimes just kinda miss the eye. I don't know if you know the movie Airplane? Yes. Of course. Where he says, I have a drinking problem. And that he keeps missing his face with the drink. That's me in the contact lens. Surely you must know that I know the movie Airplane. I do. I do know that. Stop calling me Shirley. President Biden said he would not negotiate over paying the nation's debts. But he is meeting today with House Speaker Kevin McCarthy. Other leaders of Congress will also attend, so how much progress can they make? I'm Ian Martinez with Steve Inskeep, and this is Up First from NPR News. Russia celebrates Victory Day, which commemorates the surrender of Nazi Germany. Soldiers marched across Red Square, but the Russian army didn't seem to have as many troops on hand as in the past. So what does this ritual say about the war Russia is fighting right now?"


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb


key: _node_content
string_value:
{"id_": "89cd1d2e-2159-4d67-9481-50890969616a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb", "title": "search through the reviews for a specific product", "tokens": 920, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "61bf9642-dcf0-4984-88fa-624ed03b2114", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb", "title": "search through the reviews for a specific product", "tokens": 920, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "705e49f7ddae89564d239d6f27ae0e718b4b01585ed4baf35189529de7cad20c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63d97c3b-89a6-4eb6-8a9f-3d99ecd5f970", "node_type": "1", "metadata": {}, "hash": "f204aa32899961e42dc5789573af3df436e5b6ec422254e76e9cabae1d2d02f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Semantic text search using embeddings

We can search through all our reviews semantically in a very efficient manner and at very low cost, by embedding our search query, and then finding the most similar reviews. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).


```python
import pandas as pd
import numpy as np
from ast import literal_eval

datafile_path = "data/fine_food_reviews_with_embeddings_1k.csv"

df = pd.read_csv(datafile_path)
df["embedding"] = df.embedding.apply(literal_eval).apply(np.array)

```

Here we compare the cosine similarity of the embeddings of the query and the documents, and show top_n best matches.


```python
from utils.embeddings_utils import get_embedding, cosine_similarity

# search through the reviews for a specific product
def search_reviews(df, product_description, n=3, pprint=True):
    product_embedding = get_embedding(
        product_description,
        model="text-embedding-3-small"
    )
    df["similarity"] = df.embedding.apply(lambda x: cosine_similarity(x, product_embedding))

    results = (
        df.sort_values("similarity", ascending=False)
        .head(n)
        .combined.str.replace("Title: ", "")
        .str.replace("; Content:", ": ")
    )
    if pprint:
        for r in results:
            print(r[:200])
            print()
    return results


results = search_reviews(df, "delicious beans", n=3)

```

    Delicious!:  I enjoy this white beans seasoning, it gives a rich flavor to the beans I just love it, my mother in law didn't know about this Zatarain's brand and now she is traying different seasoning
    
    Fantastic Instant Refried beans:  Fantastic Instant Refried Beans have been a staple for my family now for nearly 20 years.  All 7 of us love it and my grown kids are passing on the tradition.
    
    Delicious:  While there may be better coffee beans available, this is my first purchase and my first time grinding my own beans.  I read several reviews before purchasing this brand, and am extremely 
    



```python
results = search_reviews(df, "whole wheat pasta", n=3)

```

    Tasty and Quick Pasta:  Barilla Whole Grain Fusilli with Vegetable Marinara is tasty and has an excellent chunky vegetable marinara.  I just wish there was more of it.  If you aren't starving or on a 
    
    sooo good:  tastes so good. Worth the money. My boyfriend hates wheat pasta and LOVES this. cooks fast tastes great.I love this brand and started buying more of their pastas. Bulk is best.
    
    Bland and vaguely gamy tasting, skip this one:  As far as prepared dinner kits go, "Barilla Whole Grain Mezze Penne with Tomato and Basil Sauce" just did not do it for me...and this is coming from a p
    


We can search through these reviews easily. To speed up computation, we can use a special algorithm, aimed at faster search through embeddings.


```python
results = search_reviews(df, "bad delivery", n=1)

```

    great product, poor delivery:  The coffee is excellent and I am a repeat buyer.  Problem this time was with the UPS delivery.  They left the box in front of my garage door in the middle of the drivewa
    


As we can see, this can immediately deliver a lot of value. In this example we show being able to quickly find the examples of delivery failures.


key: _node_content
string_value:
{"id_": "63d97c3b-89a6-4eb6-8a9f-3d99ecd5f970", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb", "title": "search through the reviews for a specific product", "tokens": 920, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "61bf9642-dcf0-4984-88fa-624ed03b2114", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb", "title": "search through the reviews for a specific product", "tokens": 920, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "705e49f7ddae89564d239d6f27ae0e718b4b01585ed4baf35189529de7cad20c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89cd1d2e-2159-4d67-9481-50890969616a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb", "title": "search through the reviews for a specific product", "tokens": 920, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7419db95e725a5af0dd7d0e2066e8909c3c70099d1bb9743b9a9940888a6dfa3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2064, "end_char_idx": 4060, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results = search_reviews(df, "whole wheat pasta", n=3)

```

    Tasty and Quick Pasta:  Barilla Whole Grain Fusilli with Vegetable Marinara is tasty and has an excellent chunky vegetable marinara.  I just wish there was more of it.  If you aren't starving or on a 
    
    sooo good:  tastes so good. Worth the money. My boyfriend hates wheat pasta and LOVES this. cooks fast tastes great.I love this brand and started buying more of their pastas. Bulk is best.
    
    Bland and vaguely gamy tasting, skip this one:  As far as prepared dinner kits go, "Barilla Whole Grain Mezze Penne with Tomato and Basil Sauce" just did not do it for me...and this is coming from a p
    


We can search through these reviews easily. To speed up computation, we can use a special algorithm, aimed at faster search through embeddings.


```python
results = search_reviews(df, "bad delivery", n=1)

```

    great product, poor delivery:  The coffee is excellent and I am a repeat buyer.  Problem this time was with the UPS delivery.  They left the box in front of my garage door in the middle of the drivewa
    


As we can see, this can immediately deliver a lot of value. In this example we show being able to quickly find the examples of delivery failures.


```python
results = search_reviews(df, "spoilt", n=1)

```

    Disappointed:  The metal cover has severely disformed. And most of the cookies inside have been crushed into small pieces. Shopping experience is awful. I'll never buy it online again.
    



```python
results = search_reviews(df, "pet food", n=2)

```

    Great food!:  I wanted a food for a a dog with skin problems. His skin greatly improved with the switch, though he still itches some.  He loves the food. No recalls, American made with American ingred
    
    Great food!:  I wanted a food for a a dog with skin problems. His skin greatly improved with the switch, though he still itches some.  He loves the food. No recalls, American made with American ingred


key: _node_content
string_value:
{"id_": "5d87126b-f170-433e-a388-a12edd8e5356", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Visualizing_embeddings_in_2D.ipynb", "title": "Load the embeddings", "tokens": 511, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cd93e59a-8bab-49ed-b81d-99638f7e9cb5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Visualizing_embeddings_in_2D.ipynb", "title": "Load the embeddings", "tokens": 511, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e32422cb341115839013778ae0ce184642e4a187047f19bda45643d4663c69f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1888, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Visualizing the embeddings in 2D

We will use t-SNE to reduce the dimensionality of the embeddings from 1536 to 2. Once the embeddings are reduced to two dimensions, we can plot them in a 2D scatter plot. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).

### 1. Reduce dimensionality

We reduce the dimensionality to 2 dimensions using t-SNE decomposition.


```python
import pandas as pd
from sklearn.manifold import TSNE
import numpy as np
from ast import literal_eval

# Load the embeddings
datafile_path = "data/fine_food_reviews_with_embeddings_1k.csv"
df = pd.read_csv(datafile_path)

# Convert to a list of lists of floats
matrix = np.array(df.embedding.apply(literal_eval).to_list())

# Create a t-SNE model and transform the data
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)
vis_dims = tsne.fit_transform(matrix)
vis_dims.shape
```




    (1000, 2)



### 2. Plotting the embeddings

We colour each review by its star rating, ranging from red to green.

We can observe a decent data separation even in the reduced 2 dimensions.


```python
import matplotlib.pyplot as plt
import matplotlib
import numpy as np

colors = ["red", "darkorange", "gold", "turquoise", "darkgreen"]
x = [x for x,y in vis_dims]
y = [y for x,y in vis_dims]
color_indices = df.Score.values - 1

colormap = matplotlib.colors.ListedColormap(colors)
plt.scatter(x, y, c=color_indices, cmap=colormap, alpha=0.3)
for score in [0,1,2,3,4]:
    avg_x = np.array(x)[df.Score-1==score].mean()
    avg_y = np.array(y)[df.Score-1==score].mean()
    color = colors[score]
    plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)

plt.title("Amazon ratings visualized in language using t-SNE")
```




    Text(0.5, 1.0, 'Amazon ratings visualized in language using t-SNE')




    
![png](output_5_1.png)


key: _node_content
string_value:
{"id_": "46472345-4866-4b16-bad9-36a53370074a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fab6573-40e0-417e-89d7-648f6a678e2b", "node_type": "1", "metadata": {}, "hash": "f6e7091e81b5a59b20c9696c764977af760dd2c306fb9e22bb8a57b7fc8fc86c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2917, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Assistants API Overview (Python SDK)

The new [Assistants API](https://platform.openai.com/docs/assistants/overview) is a stateful evolution of our [Chat Completions API](https://platform.openai.com/docs/guides/text-generation/chat-completions-api) meant to simplify the creation of assistant-like experiences, and enable developer access to powerful tools like Code Interpreter and Retrieval.

![Assistants API Diagram](../images/assistants_overview_diagram.png)

## Chat Completions API vs Assistants API

The primitives of the **Chat Completions API** are `Messages`, on which you perform a `Completion` with a `Model` (`gpt-3.5-turbo`, `gpt-4`, etc). It is lightweight and powerful, but inherently stateless, which means you have to manage conversation state, tool definitions, retrieval documents, and code execution manually.

The primitives of the **Assistants API** are

- `Assistants`, which encapsulate a base model, instructions, tools, and (context) documents,
- `Threads`, which represent the state of a conversation, and
- `Runs`, which power the execution of an `Assistant` on a `Thread`, including textual responses and multi-step tool use.

We'll take a look at how these can be used to create powerful, stateful experiences.


## Setup

### Python SDK

> **Note**
> We've updated our [Python SDK](https://github.com/openai/openai-python) to add support for the Assistants API, so you'll need to update it to the latest version (`1.2.3` at time of writing).



```python
!pip install --upgrade openai
```

And make sure it's up to date by running:



```python
!pip show openai | grep Version
```

    Version: 1.2.3


### Pretty Printing Helper



```python
import json

def show_json(obj):
    display(json.loads(obj.model_dump_json()))
```

## Complete Example with Assistants API


### Assistants


The easiest way to get started with the Assistants API is through the [Assistants Playground](https://platform.openai.com/playground).


![Assistants Playground](../images/assistants_overview_assistants_playground.png)


Let's begin by creating an assistant! We'll create a Math Tutor just like in our [docs](https://platform.openai.com/docs/assistants/overview).


![Creating New Assistant](../images/assistants_overview_new_assistant.png)


You can view Assistants you've created in the [Assistants Dashboard](https://platform.openai.com/assistants).


![Assistants Dashboard](../images/assistants_overview_assistants_dashboard.png)


You can also create Assistants directly through the Assistants API, like so:



```python
from openai import OpenAI
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))


assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model="gpt-4-1106-preview",
)
show_json(assistant)
```


key: _node_content
string_value:
{"id_": "2fab6573-40e0-417e-89d7-648f6a678e2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46472345-4866-4b16-bad9-36a53370074a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "79bc04079513f5cb47b87ea7fa72ab3cef954057fa4cff4a6dc0964a513be4dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90243f91-1088-4794-a7c1-1a28ba386724", "node_type": "1", "metadata": {}, "hash": "2e7ad6749b0ddfa89326afbd53e65833a2dcf1359479427e4b756a7456881300", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1480, "end_char_idx": 4128, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install --upgrade openai
```

And make sure it's up to date by running:



```python
!pip show openai | grep Version
```

    Version: 1.2.3


### Pretty Printing Helper



```python
import json

def show_json(obj):
    display(json.loads(obj.model_dump_json()))
```

## Complete Example with Assistants API


### Assistants


The easiest way to get started with the Assistants API is through the [Assistants Playground](https://platform.openai.com/playground).


![Assistants Playground](../images/assistants_overview_assistants_playground.png)


Let's begin by creating an assistant! We'll create a Math Tutor just like in our [docs](https://platform.openai.com/docs/assistants/overview).


![Creating New Assistant](../images/assistants_overview_new_assistant.png)


You can view Assistants you've created in the [Assistants Dashboard](https://platform.openai.com/assistants).


![Assistants Dashboard](../images/assistants_overview_assistants_dashboard.png)


You can also create Assistants directly through the Assistants API, like so:



```python
from openai import OpenAI
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))


assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model="gpt-4-1106-preview",
)
show_json(assistant)
```


    {'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': []}


Regardless of whether you create your Assistant through the Dashboard or with the API, you'll want to keep track of the Assistant ID. This is how you'll refer to your Assistant throughout Threads and Runs.


Next, we'll create a new Thread and add a Message to it. This will hold the state of our conversation, so we don't have re-send the entire message history each time.


### Threads


Create a new thread:



```python
thread = client.beta.threads.create()
show_json(thread)
```


    {'id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'created_at': 1699828331,
     'metadata': {},
     'object': 'thread'}


Then add the Message to the thread:



```python
message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="I need to solve the equation `3x + 11 = 14`. Can you help me?",
)
show_json(message)
```


key: _node_content
string_value:
{"id_": "90243f91-1088-4794-a7c1-1a28ba386724", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fab6573-40e0-417e-89d7-648f6a678e2b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2d7d295f598543ee639ad5a2cec689dba8bac2134d3838ecf8729ea691390ded", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0eec7b41-2c60-430f-8ba6-4b49c720ecb2", "node_type": "1", "metadata": {}, "hash": "9e58a03dd29c0ffaba2cd1091e32a500edcf38e807728e79c5de02cd4915e3ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2693, "end_char_idx": 5688, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model="gpt-4-1106-preview",
)
show_json(assistant)
```


    {'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': []}


Regardless of whether you create your Assistant through the Dashboard or with the API, you'll want to keep track of the Assistant ID. This is how you'll refer to your Assistant throughout Threads and Runs.


Next, we'll create a new Thread and add a Message to it. This will hold the state of our conversation, so we don't have re-send the entire message history each time.


### Threads


Create a new thread:



```python
thread = client.beta.threads.create()
show_json(thread)
```


    {'id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'created_at': 1699828331,
     'metadata': {},
     'object': 'thread'}


Then add the Message to the thread:



```python
message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="I need to solve the equation `3x + 11 = 14`. Can you help me?",
)
show_json(message)
```


    {'id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
     'assistant_id': None,
     'content': [{'text': {'annotations': [],
        'value': 'I need to solve the equation `3x + 11 = 14`. Can you help me?'},
       'type': 'text'}],
     'created_at': 1699828332,
     'file_ids': [],
     'metadata': {},
     'object': 'thread.message',
     'role': 'user',
     'run_id': None,
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'}


> **Note**
> Even though you're no longer sending the entire history each time, you will still be charged for the tokens of the entire conversation history with each Run.


### Runs

Notice how the Thread we created is **not** associated with the Assistant we created earlier! Threads exist independently from Assistants, which may be different from what you'd expect if you've used ChatGPT (where a thread is tied to a model/GPT).

To get a completion from an Assistant for a given Thread, we must create a Run. Creating a Run will indicate to an Assistant it should look at the messages in the Thread and take action: either by adding a single response, or using tools.

> **Note**
> Runs are a key difference between the Assistants API and Chat Completions API. While in Chat Completions the model will only ever respond with a single message, in the Assistants API a Run may result in an Assistant using one or multiple tools, and potentially adding multiple messages to the Thread.

To get our Assistant to respond to the user, let's create the Run. As mentioned earlier, you must specify _both_ the Assistant and the Thread.


key: _node_content
string_value:
{"id_": "0eec7b41-2c60-430f-8ba6-4b49c720ecb2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90243f91-1088-4794-a7c1-1a28ba386724", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8687f084fbf35942240c244e87dd3e34c4d0e882589b01c9257a9efbb21ed39a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a01cb9f-29f2-4716-bb4d-1cec92136a89", "node_type": "1", "metadata": {}, "hash": "f11504b0b00ebeb5f7a032d5c05d76c3acc2cb130151a595de24886a9b54f81a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4135, "end_char_idx": 6443, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
     'assistant_id': None,
     'content': [{'text': {'annotations': [],
        'value': 'I need to solve the equation `3x + 11 = 14`. Can you help me?'},
       'type': 'text'}],
     'created_at': 1699828332,
     'file_ids': [],
     'metadata': {},
     'object': 'thread.message',
     'role': 'user',
     'run_id': None,
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'}


> **Note**
> Even though you're no longer sending the entire history each time, you will still be charged for the tokens of the entire conversation history with each Run.


### Runs

Notice how the Thread we created is **not** associated with the Assistant we created earlier! Threads exist independently from Assistants, which may be different from what you'd expect if you've used ChatGPT (where a thread is tied to a model/GPT).

To get a completion from an Assistant for a given Thread, we must create a Run. Creating a Run will indicate to an Assistant it should look at the messages in the Thread and take action: either by adding a single response, or using tools.

> **Note**
> Runs are a key difference between the Assistants API and Chat Completions API. While in Chat Completions the model will only ever respond with a single message, in the Assistants API a Run may result in an Assistant using one or multiple tools, and potentially adding multiple messages to the Thread.

To get our Assistant to respond to the user, let's create the Run. As mentioned earlier, you must specify _both_ the Assistant and the Thread.



```python
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
)
show_json(run)
```


    {'id': 'run_LA08RjouV3RemQ78UZXuyzv6',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': None,
     'created_at': 1699828332,
     'expires_at': 1699828932,
     'failed_at': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': None,
     'started_at': None,
     'status': 'queued',
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'tools': []}


key: _node_content
string_value:
{"id_": "1a01cb9f-29f2-4716-bb4d-1cec92136a89", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0eec7b41-2c60-430f-8ba6-4b49c720ecb2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1e414d4c64666909100f0a9c177b179045ee45bf51e1dabd0c2500622234cd2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1da5a0de-da6c-421f-b2f3-4e11bf6aba53", "node_type": "1", "metadata": {}, "hash": "833165d48d78f712073369f6807c4c944a14ff831e889f35dc4857a97ee4a1df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5692, "end_char_idx": 8421, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
)
show_json(run)
```


    {'id': 'run_LA08RjouV3RemQ78UZXuyzv6',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': None,
     'created_at': 1699828332,
     'expires_at': 1699828932,
     'failed_at': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': None,
     'started_at': None,
     'status': 'queued',
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'tools': []}


Unlike creating a completion in the Chat Completions API, **creating a Run is an asynchronous operation**. It will return immediately with the Run's metadata, which includes a `status` that will initially be set to `queued`. The `status` will be updated as the Assistant performs operations (like using tools and adding messages).

To know when the Assistant has completed processing, we can poll the Run in a loop. (Support for streaming is coming soon!) While here we are only checking for a `queued` or `in_progress` status, in practice a Run may undergo a [variety of status changes](https://platform.openai.com/docs/api-reference/runs/object#runs/object-status) which you can choose to surface to the user. (These are called Steps, and will be covered later.)



```python
import time

def wait_on_run(run, thread):
    while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=thread.id,
            run_id=run.id,
        )
        time.sleep(0.5)
    return run
```


```python
run = wait_on_run(run, thread)
show_json(run)
```


    {'id': 'run_LA08RjouV3RemQ78UZXuyzv6',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': 1699828333,
     'created_at': 1699828332,
     'expires_at': None,
     'failed_at': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': None,
     'started_at': 1699828332,
     'status': 'completed',
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'tools': []}


### Messages


Now that the Run has completed, we can list the Messages in the Thread to see what got added by the Assistant.



```python
messages = client.beta.threads.messages.list(thread_id=thread.id)
show_json(messages)
```


key: _node_content
string_value:
{"id_": "1da5a0de-da6c-421f-b2f3-4e11bf6aba53", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a01cb9f-29f2-4716-bb4d-1cec92136a89", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "350c8c6e4cb3f66f04ace069045510fa8839ad7a4318554d3e941c7aef0fa4af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "71b1b64f-628e-4c20-9023-99a13737f941", "node_type": "1", "metadata": {}, "hash": "8af1f3a49a34301bc738ffce998ecef902e4944b80d66c40cadc6169f09cc287", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7214, "end_char_idx": 9556, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import time

def wait_on_run(run, thread):
    while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=thread.id,
            run_id=run.id,
        )
        time.sleep(0.5)
    return run
```


```python
run = wait_on_run(run, thread)
show_json(run)
```


    {'id': 'run_LA08RjouV3RemQ78UZXuyzv6',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': 1699828333,
     'created_at': 1699828332,
     'expires_at': None,
     'failed_at': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': None,
     'started_at': 1699828332,
     'status': 'completed',
     'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG',
     'tools': []}


### Messages


Now that the Run has completed, we can list the Messages in the Thread to see what got added by the Assistant.



```python
messages = client.beta.threads.messages.list(thread_id=thread.id)
show_json(messages)
```


    {'data': [{'id': 'msg_S0ZtKIWjyWtbIW9JNUocPdUS',
       'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
       'content': [{'text': {'annotations': [],
          'value': 'Yes. Subtract 11 from both sides to get `3x = 3`, then divide by 3 to find `x = 1`.'},
         'type': 'text'}],
       'created_at': 1699828333,
       'file_ids': [],
       'metadata': {},
       'object': 'thread.message',
       'role': 'assistant',
       'run_id': 'run_LA08RjouV3RemQ78UZXuyzv6',
       'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'},
      {'id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
       'assistant_id': None,
       'content': [{'text': {'annotations': [],
          'value': 'I need to solve the equation `3x + 11 = 14`. Can you help me?'},
         'type': 'text'}],
       'created_at': 1699828332,
       'file_ids': [],
       'metadata': {},
       'object': 'thread.message',
       'role': 'user',
       'run_id': None,
       'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'}],
     'object': 'list',
     'first_id': 'msg_S0ZtKIWjyWtbIW9JNUocPdUS',
     'last_id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
     'has_more': False}


key: _node_content
string_value:
{"id_": "71b1b64f-628e-4c20-9023-99a13737f941", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1da5a0de-da6c-421f-b2f3-4e11bf6aba53", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efc56df99a91e2ea88f1b6d41579e83a7ce8cbfc3fd8fd9d255f0137f4a076fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "643cc821-7115-481d-9c89-5c1a088e2bfc", "node_type": "1", "metadata": {}, "hash": "158e8b1347dc08c8e963b05e2e2b68a609bab9ccab5965026eb40f09db5f1209", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8428, "end_char_idx": 10437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'data': [{'id': 'msg_S0ZtKIWjyWtbIW9JNUocPdUS',
       'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
       'content': [{'text': {'annotations': [],
          'value': 'Yes. Subtract 11 from both sides to get `3x = 3`, then divide by 3 to find `x = 1`.'},
         'type': 'text'}],
       'created_at': 1699828333,
       'file_ids': [],
       'metadata': {},
       'object': 'thread.message',
       'role': 'assistant',
       'run_id': 'run_LA08RjouV3RemQ78UZXuyzv6',
       'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'},
      {'id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
       'assistant_id': None,
       'content': [{'text': {'annotations': [],
          'value': 'I need to solve the equation `3x + 11 = 14`. Can you help me?'},
         'type': 'text'}],
       'created_at': 1699828332,
       'file_ids': [],
       'metadata': {},
       'object': 'thread.message',
       'role': 'user',
       'run_id': None,
       'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'}],
     'object': 'list',
     'first_id': 'msg_S0ZtKIWjyWtbIW9JNUocPdUS',
     'last_id': 'msg_IBiZDAWHhWPewxzN0EfTYNew',
     'has_more': False}


As you can see, Messages are ordered in reverse-chronological order – this was done so the most recent results are always on the first `page` (since results can be paginated). Do keep a look out for this, since this is the opposite order to messages in the Chat Completions API.


Let's ask our Assistant to explain the result a bit further!



```python
# Create a message to append to our thread
message = client.beta.threads.messages.create(
    thread_id=thread.id, role="user", content="Could you explain this to me?"
)

# Execute our run
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
)

# Wait for completion
wait_on_run(run, thread)

# Retrieve all the messages added after our last user message
messages = client.beta.threads.messages.list(
    thread_id=thread.id, order="asc", after=message.id
)
show_json(messages)
```


key: _node_content
string_value:
{"id_": "643cc821-7115-481d-9c89-5c1a088e2bfc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71b1b64f-628e-4c20-9023-99a13737f941", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8c0f129d8773aa47760c59fa0a9325cb38fa12b847fc04e2dbd7ab5ac7f5e51b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7d70a16-350f-4270-aefc-10600dd33e6d", "node_type": "1", "metadata": {}, "hash": "9262b7845c2df47a716738c1be0d3055dff60a29d2ea3f5a2ca64c3b141e7cad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9559, "end_char_idx": 11916, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As you can see, Messages are ordered in reverse-chronological order – this was done so the most recent results are always on the first `page` (since results can be paginated). Do keep a look out for this, since this is the opposite order to messages in the Chat Completions API.


Let's ask our Assistant to explain the result a bit further!



```python
# Create a message to append to our thread
message = client.beta.threads.messages.create(
    thread_id=thread.id, role="user", content="Could you explain this to me?"
)

# Execute our run
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id,
)

# Wait for completion
wait_on_run(run, thread)

# Retrieve all the messages added after our last user message
messages = client.beta.threads.messages.list(
    thread_id=thread.id, order="asc", after=message.id
)
show_json(messages)
```


    {'data': [{'id': 'msg_9MAeOrGriHcImeQnAzvYyJbs',
       'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
       'content': [{'text': {'annotations': [],
          'value': 'Certainly. To solve for x in the equation `3x + 11 = 14`:\n\n1. Subtract 11 from both sides: `3x + 11 - 11 = 14 - 11` simplifies to `3x = 3`.\n2. Divide both sides by 3: `3x / 3 = 3 / 3` simplifies to `x = 1`.\n\nSo, the solution is `x = 1`.'},
         'type': 'text'}],
       'created_at': 1699828335,
       'file_ids': [],
       'metadata': {},
       'object': 'thread.message',
       'role': 'assistant',
       'run_id': 'run_IFHfsubkJv7RSUbDZpNVs4PG',
       'thread_id': 'thread_bw42vPoQtYBMQE84WubNcJXG'}],
     'object': 'list',
     'first_id': 'msg_9MAeOrGriHcImeQnAzvYyJbs',
     'last_id': 'msg_9MAeOrGriHcImeQnAzvYyJbs',
     'has_more': False}


This may feel like a lot of steps to get a response back, especially for this simple example. However, you'll soon see how we can add very powerful functionality to our Assistant without changing much code at all!


### Example


Let's take a look at how we could potentially put all of this together. Below is all the code you need to use an Assistant you've created.

Since we've already created our Math Assistant, I've saved its ID in `MATH_ASSISTANT_ID`. I then defined two functions:

- `submit_message`: create a Message on a Thread, then start (and return) a new Run
- `get_response`: returns the list of Messages in a Thread


key: _node_content
string_value:
{"id_": "e7d70a16-350f-4270-aefc-10600dd33e6d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "643cc821-7115-481d-9c89-5c1a088e2bfc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7ea5605056be2a436ae62a9f27d9767ff9343b17ebcacc58bdce1e525f524eba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5332c196-6a8f-4a71-b48c-4e624d08ac83", "node_type": "1", "metadata": {}, "hash": "541caf0c4881fb0165fc2719bfb0e8acab66a7bd2b907e17703dc87e129ed116", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11283, "end_char_idx": 14083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This may feel like a lot of steps to get a response back, especially for this simple example. However, you'll soon see how we can add very powerful functionality to our Assistant without changing much code at all!


### Example


Let's take a look at how we could potentially put all of this together. Below is all the code you need to use an Assistant you've created.

Since we've already created our Math Assistant, I've saved its ID in `MATH_ASSISTANT_ID`. I then defined two functions:

- `submit_message`: create a Message on a Thread, then start (and return) a new Run
- `get_response`: returns the list of Messages in a Thread



```python
from openai import OpenAI

MATH_ASSISTANT_ID = assistant.id  # or a hard-coded ID like "asst-..."

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

def submit_message(assistant_id, thread, user_message):
    client.beta.threads.messages.create(
        thread_id=thread.id, role="user", content=user_message
    )
    return client.beta.threads.runs.create(
        thread_id=thread.id,
        assistant_id=assistant_id,
    )


def get_response(thread):
    return client.beta.threads.messages.list(thread_id=thread.id, order="asc")
```

I've also defined a `create_thread_and_run` function that I can re-use (which is actually almost identical to the [`client.beta.threads.create_and_run`](https://platform.openai.com/docs/api-reference/runs/createThreadAndRun) compound function in our API ;) ). Finally, we can submit our mock user requests each to a new Thread.

Notice how all of these API calls are asynchronous operations; this means we actually get async behavior in our code without the use of async libraries! (e.g. `asyncio`)



```python
def create_thread_and_run(user_input):
    thread = client.beta.threads.create()
    run = submit_message(MATH_ASSISTANT_ID, thread, user_input)
    return thread, run


# Emulating concurrent user requests
thread1, run1 = create_thread_and_run(
    "I need to solve the equation `3x + 11 = 14`. Can you help me?"
)
thread2, run2 = create_thread_and_run("Could you explain linear algebra to me?")
thread3, run3 = create_thread_and_run("I don't like math. What can I do?")

# Now all Runs are executing...
```

Once all Runs are going, we can wait on each and get the responses.



```python
import time

# Pretty printing helper
def pretty_print(messages):
    print("# Messages")
    for m in messages:
        print(f"{m.role}: {m.content[0].text.value}")
    print()


# Waiting in a loop
def wait_on_run(run, thread):
    while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=thread.id,
            run_id=run.id,
        )
        time.sleep(0.5)
    return run


key: _node_content
string_value:
{"id_": "5332c196-6a8f-4a71-b48c-4e624d08ac83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7d70a16-350f-4270-aefc-10600dd33e6d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "79e2a6f35cb51fd0f293dc9c1a94a3dcc50ffb3199808cacee071ac991a797d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4550d261-e3fa-453b-9244-306bc1c0cde7", "node_type": "1", "metadata": {}, "hash": "507e175905c4c0166ec8d8fa9a55444e464244427ed3d3e45325b4d8f7927200", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13029, "end_char_idx": 16193, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def create_thread_and_run(user_input):
    thread = client.beta.threads.create()
    run = submit_message(MATH_ASSISTANT_ID, thread, user_input)
    return thread, run


# Emulating concurrent user requests
thread1, run1 = create_thread_and_run(
    "I need to solve the equation `3x + 11 = 14`. Can you help me?"
)
thread2, run2 = create_thread_and_run("Could you explain linear algebra to me?")
thread3, run3 = create_thread_and_run("I don't like math. What can I do?")

# Now all Runs are executing...
```

Once all Runs are going, we can wait on each and get the responses.



```python
import time

# Pretty printing helper
def pretty_print(messages):
    print("# Messages")
    for m in messages:
        print(f"{m.role}: {m.content[0].text.value}")
    print()


# Waiting in a loop
def wait_on_run(run, thread):
    while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=thread.id,
            run_id=run.id,
        )
        time.sleep(0.5)
    return run


# Wait for Run 1
run1 = wait_on_run(run1, thread1)
pretty_print(get_response(thread1))

# Wait for Run 2
run2 = wait_on_run(run2, thread2)
pretty_print(get_response(thread2))

# Wait for Run 3
run3 = wait_on_run(run3, thread3)
pretty_print(get_response(thread3))

# Thank our assistant on Thread 3 :)
run4 = submit_message(MATH_ASSISTANT_ID, thread3, "Thank you!")
run4 = wait_on_run(run4, thread3)
pretty_print(get_response(thread3))
```

    # Messages
    user: I need to solve the equation `3x + 11 = 14`. Can you help me?
    assistant: Yes, subtract 11 from both sides to get `3x = 3`, then divide both sides by 3 to find `x = 1`.
    
    # Messages
    user: Could you explain linear algebra to me?
    assistant: Linear algebra is the branch of mathematics that deals with vector spaces, linear equations, and matrices, focusing on the properties and operations that can be applied to vectors and linear transformations.
    
    # Messages
    user: I don't like math. What can I do?
    assistant: Try finding aspects of math that relate to your interests or daily life, and consider a tutor or interactive resources to make learning more enjoyable.
    
    # Messages
    user: I don't like math. What can I do?
    assistant: Try finding aspects of math that relate to your interests or daily life, and consider a tutor or interactive resources to make learning more enjoyable.
    user: Thank you!
    assistant: You're welcome! If you have any more questions, feel free to ask.
    


Et voilà!

You may have noticed that this code is not actually specific to our math Assistant at all... this code will work for any new Assistant you create simply by changing the Assistant ID! That is the power of the Assistants API.


## Tools

A key feature of the Assistants API is the ability to equip our Assistants with Tools, like Code Interpreter, Retrieval, and custom Functions. Let's take a look at each.

### Code Interpreter

Let's equip our Math Tutor with the [Code Interpreter](https://platform.openai.com/docs/assistants/tools/code-interpreter) tool, which we can do from the Dashboard...


key: _node_content
string_value:
{"id_": "4550d261-e3fa-453b-9244-306bc1c0cde7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5332c196-6a8f-4a71-b48c-4e624d08ac83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "3cd8db29f825631cd1b5d7fd63d0dc9b113d54fb8b0ceb531acd983d11fea928", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df422cf4-e5b2-4663-9554-7f697a18c991", "node_type": "1", "metadata": {}, "hash": "2810bf5ba0c1bd0af8e0417b678bb1782a379df96fd27220f392b9dd4c25e18d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15587, "end_char_idx": 18122, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Et voilà!

You may have noticed that this code is not actually specific to our math Assistant at all... this code will work for any new Assistant you create simply by changing the Assistant ID! That is the power of the Assistants API.


## Tools

A key feature of the Assistants API is the ability to equip our Assistants with Tools, like Code Interpreter, Retrieval, and custom Functions. Let's take a look at each.

### Code Interpreter

Let's equip our Math Tutor with the [Code Interpreter](https://platform.openai.com/docs/assistants/tools/code-interpreter) tool, which we can do from the Dashboard...


![Enabling code interpreter](../images/assistants_overview_enable_code_interpreter.png)


...or the API, using the Assistant ID.



```python
assistant = client.beta.assistants.update(
    MATH_ASSISTANT_ID,
    tools=[{"type": "code_interpreter"}],
)
show_json(assistant)
```


    {'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': [],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': [{'type': 'code_interpreter'}]}


Now, let's ask the Assistant to use its new tool.



```python
thread, run = create_thread_and_run(
    "Generate the first 20 fibbonaci numbers with code."
)
run = wait_on_run(run, thread)
pretty_print(get_response(thread))
```

    # Messages
    user: Generate the first 20 fibbonaci numbers with code.
    assistant: The first 20 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, and 4181.
    


And that's it! The Assistant used Code Interpreter in the background, and gave us a final response.

For some use cases this may be enough – however, if we want more details on what precisely an Assistant is doing we can take a look at a Run's Steps.

### Steps


A Run is composed of one or more Steps. Like a Run, each Step has a `status` that you can query. This is useful for surfacing the progress of a Step to a user (e.g. a spinner while the Assistant is writing code or performing retrieval).



```python
run_steps = client.beta.threads.runs.steps.list(
    thread_id=thread.id, run_id=run.id, order="asc"
)
```

Let's take a look at each Step's `step_details`.



```python
for step in run_steps.data:
    step_details = step.step_details
    print(json.dumps(show_json(step_details), indent=4))
```


key: _node_content
string_value:
{"id_": "df422cf4-e5b2-4663-9554-7f697a18c991", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4550d261-e3fa-453b-9244-306bc1c0cde7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "23fb0b319772f39599d3162986e07462995e1bf25701e0888206ade95b0ae5f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db906cd0-0f71-49b5-ad55-08550e1f0ecf", "node_type": "1", "metadata": {}, "hash": "c589620c30040dec080c74333c4a541a0f878080e5c249d2fc0382f711dea259", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16861, "end_char_idx": 19460, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now, let's ask the Assistant to use its new tool.



```python
thread, run = create_thread_and_run(
    "Generate the first 20 fibbonaci numbers with code."
)
run = wait_on_run(run, thread)
pretty_print(get_response(thread))
```

    # Messages
    user: Generate the first 20 fibbonaci numbers with code.
    assistant: The first 20 Fibonacci numbers are: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, and 4181.
    


And that's it! The Assistant used Code Interpreter in the background, and gave us a final response.

For some use cases this may be enough – however, if we want more details on what precisely an Assistant is doing we can take a look at a Run's Steps.

### Steps


A Run is composed of one or more Steps. Like a Run, each Step has a `status` that you can query. This is useful for surfacing the progress of a Step to a user (e.g. a spinner while the Assistant is writing code or performing retrieval).



```python
run_steps = client.beta.threads.runs.steps.list(
    thread_id=thread.id, run_id=run.id, order="asc"
)
```

Let's take a look at each Step's `step_details`.



```python
for step in run_steps.data:
    step_details = step.step_details
    print(json.dumps(show_json(step_details), indent=4))
```


    {'tool_calls': [{'id': 'call_WMNqd63PtX8vZzTwaA6eWpBg',
       'code_interpreter': {'input': '# Python function to generate the first 20 Fibonacci numbers\ndef fibonacci(n):\n    fib_sequence = [0, 1]\n    while len(fib_sequence) < n:\n        fib_sequence.append(fib_sequence[-1] + fib_sequence[-2])\n    return fib_sequence\n\n# Generate the first 20 Fibonacci numbers\nfirst_20_fibonacci = fibonacci(20)\nfirst_20_fibonacci',
        'outputs': [{'logs': '[0,\n 1,\n 1,\n 2,\n 3,\n 5,\n 8,\n 13,\n 21,\n 34,\n 55,\n 89,\n 144,\n 233,\n 377,\n 610,\n 987,\n 1597,\n 2584,\n 4181]',
          'type': 'logs'}]},
       'type': 'code_interpreter'}],
     'type': 'tool_calls'}


    null



    {'message_creation': {'message_id': 'msg_z593lE5bvcD6BngeDFHDxzwm'},
     'type': 'message_creation'}


    null


We can see the `step_details` for two Steps:

1. `tool_calls` (plural, since it could be more than one in a single Step)
2. `message_creation`

The first Step is a `tool_calls`, specifically using the `code_interpreter` which contains:

- `input`, which was the Python code generated before the tool was called, and
- `output`, which was the result of running the Code Interpreter.

The second Step is a `message_creation`, which contains the `message` that was added to the Thread to communicate the results to the user.


key: _node_content
string_value:
{"id_": "db906cd0-0f71-49b5-ad55-08550e1f0ecf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df422cf4-e5b2-4663-9554-7f697a18c991", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cd3b3d697f8f5ff1e671ce63f93e787af1e0bf61c07eb7f64b857f379a5af437", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae66295d-9fee-4af8-bb45-0f49d10c795e", "node_type": "1", "metadata": {}, "hash": "a20d0558bba64f77eaedd7089f31ec68e2642e1b9efa904aa2656339115bb86e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18812, "end_char_idx": 22081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
null



    {'message_creation': {'message_id': 'msg_z593lE5bvcD6BngeDFHDxzwm'},
     'type': 'message_creation'}


    null


We can see the `step_details` for two Steps:

1. `tool_calls` (plural, since it could be more than one in a single Step)
2. `message_creation`

The first Step is a `tool_calls`, specifically using the `code_interpreter` which contains:

- `input`, which was the Python code generated before the tool was called, and
- `output`, which was the result of running the Code Interpreter.

The second Step is a `message_creation`, which contains the `message` that was added to the Thread to communicate the results to the user.


### Retrieval

Another powerful tool in the Assistants API is [Retrieval](https://platform.openai.com/docs/assistants/tools/knowledge-retrieval): the ability to upload files that the Assistant will use as a knowledge base when answering questions. This can also be enabled from the Dashboard or the API, where we can upload files we want to be used.


![Enabling retrieval](../images/assistants_overview_enable_retrieval.png)



```python
# Upload the file
file = client.files.create(
    file=open(
        "data/language_models_are_unsupervised_multitask_learners.pdf",
        "rb",
    ),
    purpose="assistants",
)
# Update Assistant
assistant = client.beta.assistants.update(
    MATH_ASSISTANT_ID,
    tools=[{"type": "code_interpreter"}, {"type": "retrieval"}],
    file_ids=[file.id],
)
show_json(assistant)
```


    {'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': ['file-MdXcQI8OdPp76wukWI4dpLwW'],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': [{'type': 'code_interpreter'}, {'type': 'retrieval'}]}



```python
thread, run = create_thread_and_run(
    "What are some cool math concepts behind this ML paper pdf? Explain in two sentences."
)
run = wait_on_run(run, thread)
pretty_print(get_response(thread))
```

    # Messages
    user: What are some cool math concepts behind this ML paper pdf? Explain in two sentences.
    assistant: I am unable to find specific sections referring to "cool math concepts" directly in the paper using the available tools. I will now read the beginning of the paper to identify any mathematical concepts that are fundamental to the paper.
    assistant: The paper discusses leveraging large language models as a framework for unsupervised multitask learning, where tasks are implicitly defined by the context within sequences of text. It explores the zero-shot learning capabilities of such models by showing that when a language model is trained on a vast dataset, it begins to generalize and perform tasks without explicit supervision, achieving competitive results across various natural language processing tasks using a probabilistic framework based on sequential modeling and conditional probabilities.
    


> **Note**
> There are more intricacies in Retrieval, like [Annotations](https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages), which may be covered in another cookbook.


key: _node_content
string_value:
{"id_": "ae66295d-9fee-4af8-bb45-0f49d10c795e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db906cd0-0f71-49b5-ad55-08550e1f0ecf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d7de947916510e99dc696ca6d6ff3463cee5d3a9780fa1d0313f4e3b6865ee2c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b42ce736-304d-4b73-8d82-27eaff924319", "node_type": "1", "metadata": {}, "hash": "5ddbdf30fed006677bb110c3e1f3563b26a262fedbbf562207c274e3453cac74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20728, "end_char_idx": 23989, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
thread, run = create_thread_and_run(
    "What are some cool math concepts behind this ML paper pdf? Explain in two sentences."
)
run = wait_on_run(run, thread)
pretty_print(get_response(thread))
```

    # Messages
    user: What are some cool math concepts behind this ML paper pdf? Explain in two sentences.
    assistant: I am unable to find specific sections referring to "cool math concepts" directly in the paper using the available tools. I will now read the beginning of the paper to identify any mathematical concepts that are fundamental to the paper.
    assistant: The paper discusses leveraging large language models as a framework for unsupervised multitask learning, where tasks are implicitly defined by the context within sequences of text. It explores the zero-shot learning capabilities of such models by showing that when a language model is trained on a vast dataset, it begins to generalize and perform tasks without explicit supervision, achieving competitive results across various natural language processing tasks using a probabilistic framework based on sequential modeling and conditional probabilities.
    


> **Note**
> There are more intricacies in Retrieval, like [Annotations](https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages), which may be covered in another cookbook.


### Functions

As a final powerful tool for your Assistant, you can specify custom [Functions](https://platform.openai.com/docs/assistants/tools/function-calling) (much like the [Function Calling](https://platform.openai.com/docs/guides/function-calling) in the Chat Completions API). During a Run, the Assistant can then indicate it wants to call one or more functions you specified. You are then responsible for calling the Function, and providing the output back to the Assistant.

Let's take a look at an example by defining a `display_quiz()` Function for our Math Tutor.

This function will take a `title` and an array of `question`s, display the quiz, and get input from the user for each:

- `title`
- `questions`
  - `question_text`
  - `question_type`: [`MULTIPLE_CHOICE`, `FREE_RESPONSE`]
  - `choices`: ["choice 1", "choice 2", ...]

Unfortunately I don't know how to get user input within a Python Notebook, so I'll be mocking out responses with `get_mock_response...`. This is where you'd get the user's actual input.



```python
def get_mock_response_from_user_multiple_choice():
    return "a"


def get_mock_response_from_user_free_response():
    return "I don't know."


def display_quiz(title, questions):
    print("Quiz:", title)
    print()
    responses = []

    for q in questions:
        print(q["question_text"])
        response = ""

        # If multiple choice, print options
        if q["question_type"] == "MULTIPLE_CHOICE":
            for i, choice in enumerate(q["choices"]):
                print(f"{i}. {choice}")
            response = get_mock_response_from_user_multiple_choice()

        # Otherwise, just get response
        elif q["question_type"] == "FREE_RESPONSE":
            response = get_mock_response_from_user_free_response()

        responses.append(response)
        print()

    return responses
```

Here's what a sample quiz would look like:


key: _node_content
string_value:
{"id_": "b42ce736-304d-4b73-8d82-27eaff924319", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae66295d-9fee-4af8-bb45-0f49d10c795e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fa589c941a041bae5c19958cd99516ab99923069d4a49647e1a4ca3826436ede", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c156712-d071-462f-a814-3d5808b4f744", "node_type": "1", "metadata": {}, "hash": "0b5a13fde9142aa46916cad94d6337478d4c43af3c28de91d026f0facb565e97", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23119, "end_char_idx": 26354, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_mock_response_from_user_multiple_choice():
    return "a"


def get_mock_response_from_user_free_response():
    return "I don't know."


def display_quiz(title, questions):
    print("Quiz:", title)
    print()
    responses = []

    for q in questions:
        print(q["question_text"])
        response = ""

        # If multiple choice, print options
        if q["question_type"] == "MULTIPLE_CHOICE":
            for i, choice in enumerate(q["choices"]):
                print(f"{i}. {choice}")
            response = get_mock_response_from_user_multiple_choice()

        # Otherwise, just get response
        elif q["question_type"] == "FREE_RESPONSE":
            response = get_mock_response_from_user_free_response()

        responses.append(response)
        print()

    return responses
```

Here's what a sample quiz would look like:



```python
responses = display_quiz(
    "Sample Quiz",
    [
        {"question_text": "What is your name?", "question_type": "FREE_RESPONSE"},
        {
            "question_text": "What is your favorite color?",
            "question_type": "MULTIPLE_CHOICE",
            "choices": ["Red", "Blue", "Green", "Yellow"],
        },
    ],
)
print("Responses:", responses)
```

    Quiz: Sample Quiz
    
    What is your name?
    
    What is your favorite color?
    0. Red
    1. Blue
    2. Green
    3. Yellow
    
    Responses: ["I don't know.", 'a']


Now, let's define the interface of this function in JSON format, so our Assistant can call it:



```python
function_json = {
    "name": "display_quiz",
    "description": "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions.",
    "parameters": {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "questions": {
                "type": "array",
                "description": "An array of questions, each with a title and potentially options (if multiple choice).",
                "items": {
                    "type": "object",
                    "properties": {
                        "question_text": {"type": "string"},
                        "question_type": {
                            "type": "string",
                            "enum": ["MULTIPLE_CHOICE", "FREE_RESPONSE"],
                        },
                        "choices": {"type": "array", "items": {"type": "string"}},
                    },
                    "required": ["question_text"],
                },
            },
        },
        "required": ["title", "questions"],
    },
}
```

Once again, let's update our Assistant either through the Dashboard or the API.


![Enabling custom function](../images/assistants_overview_enable_function.png)

> **Note**
> Pasting the function JSON into the Dashboard was a bit finicky due to indentation, etc. I just asked ChatGPT to format my function the same as one of the examples on the Dashboard :).



```python
assistant = client.beta.assistants.update(
    MATH_ASSISTANT_ID,
    tools=[
        {"type": "code_interpreter"},
        {"type": "retrieval"},
        {"type": "function", "function": function_json},
    ],
)
show_json(assistant)
```


key: _node_content
string_value:
{"id_": "6c156712-d071-462f-a814-3d5808b4f744", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b42ce736-304d-4b73-8d82-27eaff924319", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4c35a3f5016e943214ba4502afca86e8dcc86d1fad72d6562c2ee55a8a0a1952", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0ff2d88-dc36-4deb-9ab9-bcf0950cfee2", "node_type": "1", "metadata": {}, "hash": "c8d9d2a704246f6613bdc8cb87ae43b22125cdf2ce06c982bcaa6e8a986f9187", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24554, "end_char_idx": 27669, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now, let's define the interface of this function in JSON format, so our Assistant can call it:



```python
function_json = {
    "name": "display_quiz",
    "description": "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions.",
    "parameters": {
        "type": "object",
        "properties": {
            "title": {"type": "string"},
            "questions": {
                "type": "array",
                "description": "An array of questions, each with a title and potentially options (if multiple choice).",
                "items": {
                    "type": "object",
                    "properties": {
                        "question_text": {"type": "string"},
                        "question_type": {
                            "type": "string",
                            "enum": ["MULTIPLE_CHOICE", "FREE_RESPONSE"],
                        },
                        "choices": {"type": "array", "items": {"type": "string"}},
                    },
                    "required": ["question_text"],
                },
            },
        },
        "required": ["title", "questions"],
    },
}
```

Once again, let's update our Assistant either through the Dashboard or the API.


![Enabling custom function](../images/assistants_overview_enable_function.png)

> **Note**
> Pasting the function JSON into the Dashboard was a bit finicky due to indentation, etc. I just asked ChatGPT to format my function the same as one of the examples on the Dashboard :).



```python
assistant = client.beta.assistants.update(
    MATH_ASSISTANT_ID,
    tools=[
        {"type": "code_interpreter"},
        {"type": "retrieval"},
        {"type": "function", "function": function_json},
    ],
)
show_json(assistant)
```


    {'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': ['file-MdXcQI8OdPp76wukWI4dpLwW'],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': [{'type': 'code_interpreter'},
      {'type': 'retrieval'},
      {'function': {'name': 'display_quiz',
        'parameters': {'type': 'object',
         'properties': {'title': {'type': 'string'},
          'questions': {'type': 'array',
           'description': 'An array of questions, each with a title and potentially options (if multiple choice).',
           'items': {'type': 'object',
            'properties': {'question_text': {'type': 'string'},
             'question_type': {'type': 'string',
              'enum': ['MULTIPLE_CHOICE', 'FREE_RESPONSE']},
             'choices': {'type': 'array', 'items': {'type': 'string'}}},
            'required': ['question_text']}}},
         'required': ['title', 'questions']},
        'description': "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions."},
       'type': 'function'}]}


And now, we ask for a quiz.


key: _node_content
string_value:
{"id_": "a0ff2d88-dc36-4deb-9ab9-bcf0950cfee2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c156712-d071-462f-a814-3d5808b4f744", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6cd51fc0692c108e773755ccc4471c5c3cc8d355815657393088023f61d82ffb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5b585e73-10fd-49d7-93bb-1f536943c700", "node_type": "1", "metadata": {}, "hash": "370f8ee88404904a36ac88f5639395635a9545b98b28aa0723f32acd1940d241", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26361, "end_char_idx": 28037, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'created_at': 1699828331,
     'description': None,
     'file_ids': ['file-MdXcQI8OdPp76wukWI4dpLwW'],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'name': 'Math Tutor',
     'object': 'assistant',
     'tools': [{'type': 'code_interpreter'},
      {'type': 'retrieval'},
      {'function': {'name': 'display_quiz',
        'parameters': {'type': 'object',
         'properties': {'title': {'type': 'string'},
          'questions': {'type': 'array',
           'description': 'An array of questions, each with a title and potentially options (if multiple choice).',
           'items': {'type': 'object',
            'properties': {'question_text': {'type': 'string'},
             'question_type': {'type': 'string',
              'enum': ['MULTIPLE_CHOICE', 'FREE_RESPONSE']},
             'choices': {'type': 'array', 'items': {'type': 'string'}}},
            'required': ['question_text']}}},
         'required': ['title', 'questions']},
        'description': "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions."},
       'type': 'function'}]}


And now, we ask for a quiz.



```python
thread, run = create_thread_and_run(
    "Make a quiz with 2 questions: One open ended, one multiple choice. Then, give me feedback for the responses."
)
run = wait_on_run(run, thread)
run.status
```




    'requires_action'



Now, however, when we check the Run's `status` we see `requires_action`! Let's take a closer.



```python
show_json(run)
```


key: _node_content
string_value:
{"id_": "5b585e73-10fd-49d7-93bb-1f536943c700", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0ff2d88-dc36-4deb-9ab9-bcf0950cfee2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d87c9914f826e7ee68f09b3542df1546271af9fbab0c54a0bb66f6b501c4629b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26349fc3-1fa9-46ed-96a6-a8bc010ae143", "node_type": "1", "metadata": {}, "hash": "7ee548e2857cda2d1206d9f36951766c9d210335d7f661014a9ebf32479fef86", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27642, "end_char_idx": 30294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
And now, we ask for a quiz.



```python
thread, run = create_thread_and_run(
    "Make a quiz with 2 questions: One open ended, one multiple choice. Then, give me feedback for the responses."
)
run = wait_on_run(run, thread)
run.status
```




    'requires_action'



Now, however, when we check the Run's `status` we see `requires_action`! Let's take a closer.



```python
show_json(run)
```


    {'id': 'run_98PGE3qGtHoaWaCLoytyRUBf',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': None,
     'created_at': 1699828370,
     'expires_at': 1699828970,
     'failed_at': None,
     'file_ids': ['file-MdXcQI8OdPp76wukWI4dpLwW'],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': {'submit_tool_outputs': {'tool_calls': [{'id': 'call_Zf650sWT1wW4Uwbf5YeDS0VG',
         'function': {'arguments': '{\n  "title": "Mathematics Quiz",\n  "questions": [\n    {\n      "question_text": "Explain why the square root of a negative number is not a real number.",\n      "question_type": "FREE_RESPONSE"\n    },\n    {\n      "question_text": "What is the value of an angle in a regular pentagon?",\n      "choices": [\n        "72 degrees",\n        "90 degrees",\n        "108 degrees",\n        "120 degrees"\n      ],\n      "question_type": "MULTIPLE_CHOICE"\n    }\n  ]\n}',
          'name': 'display_quiz'},
         'type': 'function'}]},
      'type': 'submit_tool_outputs'},
     'started_at': 1699828370,
     'status': 'requires_action',
     'thread_id': 'thread_bICTESFvWoRdj0O0SzsosLCS',
     'tools': [{'type': 'code_interpreter'},
      {'type': 'retrieval'},
      {'function': {'name': 'display_quiz',
        'parameters': {'type': 'object',
         'properties': {'title': {'type': 'string'},
          'questions': {'type': 'array',
           'description': 'An array of questions, each with a title and potentially options (if multiple choice).',
           'items': {'type': 'object',
            'properties': {'question_text': {'type': 'string'},
             'question_type': {'type': 'string',
              'enum': ['MULTIPLE_CHOICE', 'FREE_RESPONSE']},
             'choices': {'type': 'array', 'items': {'type': 'string'}}},
            'required': ['question_text']}}},
         'required': ['title', 'questions']},
        'description': "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions."},
       'type': 'function'}]}


key: _node_content
string_value:
{"id_": "26349fc3-1fa9-46ed-96a6-a8bc010ae143", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5b585e73-10fd-49d7-93bb-1f536943c700", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6eddf04fd3bec8fce94127f9b5fa17d9481181962eb59545c1100fc374f68072", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fddbf4a7-368d-49c7-990c-dd917c2d787c", "node_type": "1", "metadata": {}, "hash": "c098098af2e6df9c3d19857f011cf496687b60f770264cd7b5981608b75b0fcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30297, "end_char_idx": 32481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `required_action` field indicates a Tool is waiting for us to run it and submit its output back to the Assistant. Specifically, the `display_quiz` function! Let's start by parsing the `name` and `arguments`.

> **Note**
> While in this case we know there is only one Tool call, in practice the Assistant may choose to call multiple tools.



```python
# Extract single tool call
tool_call = run.required_action.submit_tool_outputs.tool_calls[0]
name = tool_call.function.name
arguments = json.loads(tool_call.function.arguments)

print("Function Name:", name)
print("Function Arguments:")
arguments
```

    Function Name: display_quiz
    Function Arguments:





    {'title': 'Mathematics Quiz',
     'questions': [{'question_text': 'Explain why the square root of a negative number is not a real number.',
       'question_type': 'FREE_RESPONSE'},
      {'question_text': 'What is the value of an angle in a regular pentagon?',
       'choices': ['72 degrees', '90 degrees', '108 degrees', '120 degrees'],
       'question_type': 'MULTIPLE_CHOICE'}]}



Now let's actually call our `display_quiz` function with the arguments provided by the Assistant:



```python
responses = display_quiz(arguments["title"], arguments["questions"])
print("Responses:", responses)
```

    Quiz: Mathematics Quiz
    
    Explain why the square root of a negative number is not a real number.
    
    What is the value of an angle in a regular pentagon?
    0. 72 degrees
    1. 90 degrees
    2. 108 degrees
    3. 120 degrees
    
    Responses: ["I don't know.", 'a']


Great! (Remember these responses are the one's we mocked earlier. In reality, we'd be getting input from the back from this function call.)

Now that we have our responses, let's submit them back to the Assistant. We'll need the `tool_call` ID, found in the `tool_call` we parsed out earlier. We'll also need to encode our `list`of responses into a `str`.



```python
run = client.beta.threads.runs.submit_tool_outputs(
    thread_id=thread.id,
    run_id=run.id,
    tool_outputs=[
        {
            "tool_call_id": tool_call.id,
            "output": json.dumps(responses),
        }
    ],
)
show_json(run)
```


key: _node_content
string_value:
{"id_": "fddbf4a7-368d-49c7-990c-dd917c2d787c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26349fc3-1fa9-46ed-96a6-a8bc010ae143", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "55ddcc2e47f7688569ab3bc6e01f84236f36452ef63a67b52777281628c7b56f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f239161-84fc-46c2-bbb6-341089953960", "node_type": "1", "metadata": {}, "hash": "2a8564f417fb71aace47221f9af9c71a1d5d467ae9de197bd30084ba335b5c9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30970, "end_char_idx": 34040, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
{'title': 'Mathematics Quiz',
     'questions': [{'question_text': 'Explain why the square root of a negative number is not a real number.',
       'question_type': 'FREE_RESPONSE'},
      {'question_text': 'What is the value of an angle in a regular pentagon?',
       'choices': ['72 degrees', '90 degrees', '108 degrees', '120 degrees'],
       'question_type': 'MULTIPLE_CHOICE'}]}



Now let's actually call our `display_quiz` function with the arguments provided by the Assistant:



```python
responses = display_quiz(arguments["title"], arguments["questions"])
print("Responses:", responses)
```

    Quiz: Mathematics Quiz
    
    Explain why the square root of a negative number is not a real number.
    
    What is the value of an angle in a regular pentagon?
    0. 72 degrees
    1. 90 degrees
    2. 108 degrees
    3. 120 degrees
    
    Responses: ["I don't know.", 'a']


Great! (Remember these responses are the one's we mocked earlier. In reality, we'd be getting input from the back from this function call.)

Now that we have our responses, let's submit them back to the Assistant. We'll need the `tool_call` ID, found in the `tool_call` we parsed out earlier. We'll also need to encode our `list`of responses into a `str`.



```python
run = client.beta.threads.runs.submit_tool_outputs(
    thread_id=thread.id,
    run_id=run.id,
    tool_outputs=[
        {
            "tool_call_id": tool_call.id,
            "output": json.dumps(responses),
        }
    ],
)
show_json(run)
```


    {'id': 'run_98PGE3qGtHoaWaCLoytyRUBf',
     'assistant_id': 'asst_9HAjl9y41ufsViNcThW1EXUS',
     'cancelled_at': None,
     'completed_at': None,
     'created_at': 1699828370,
     'expires_at': 1699828970,
     'failed_at': None,
     'file_ids': ['file-MdXcQI8OdPp76wukWI4dpLwW'],
     'instructions': 'You are a personal math tutor. Answer questions briefly, in a sentence or less.',
     'last_error': None,
     'metadata': {},
     'model': 'gpt-4-1106-preview',
     'object': 'thread.run',
     'required_action': None,
     'started_at': 1699828370,
     'status': 'queued',
     'thread_id': 'thread_bICTESFvWoRdj0O0SzsosLCS',
     'tools': [{'type': 'code_interpreter'},
      {'type': 'retrieval'},
      {'function': {'name': 'display_quiz',
        'parameters': {'type': 'object',
         'properties': {'title': {'type': 'string'},
          'questions': {'type': 'array',
           'description': 'An array of questions, each with a title and potentially options (if multiple choice).',
           'items': {'type': 'object',
            'properties': {'question_text': {'type': 'string'},
             'question_type': {'type': 'string',
              'enum': ['MULTIPLE_CHOICE', 'FREE_RESPONSE']},
             'choices': {'type': 'array', 'items': {'type': 'string'}}},
            'required': ['question_text']}}},
         'required': ['title', 'questions']},
        'description': "Displays a quiz to the student, and returns the student's response. A single quiz can have multiple questions."},
       'type': 'function'}]}


key: _node_content
string_value:
{"id_": "3f239161-84fc-46c2-bbb6-341089953960", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "befc185e-7c91-493d-9ad9-719f39d32a5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "25196582ecca1a5525badb6fba4d240193b570af0cf05090a29a71b5dfc22b98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fddbf4a7-368d-49c7-990c-dd917c2d787c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Assistants_API_overview_python.ipynb", "title": "Assistants API Overview (Python SDK)", "tokens": 9318, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8d974407c62f5ec8433f1cd2793e0759d22785778758af4f561e564d0ea5f724", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34043, "end_char_idx": 36387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We can now wait for the Run to complete once again, and check our Thread!



```python
run = wait_on_run(run, thread)
pretty_print(get_response(thread))
```

    # Messages
    user: Make a quiz with 2 questions: One open ended, one multiple choice. Then, give me feedback for the responses.
    assistant: Thank you for attempting the quiz.
    
    For the first question, it's important to know that the square root of a negative number is not a real number because real numbers consist of all the numbers on the number line, and that includes all positive numbers, zero, and negative numbers. However, the square root of a negative number is not on this number line; instead, it is what we call an imaginary number. When we want to take the square root of a negative number, we typically use the imaginary unit \(i\), where \(i\) is defined as \(\sqrt{-1}\).
    
    For the second question, the correct answer is "108 degrees." In a regular pentagon, which is a five-sided polygon with equal sides and angles, each interior angle is \(108\) degrees. This is because the sum of the interior angles of a pentagon is \(540\) degrees, and when divided by \(5\) (the number of angles), it gives \(108\) degrees per angle. The choice you selected, "72 degrees," actually refers to the external angle of a regular pentagon, not the internal angle.
    


Woohoo 🎉


## Conclusion

We covered a lot of ground in this notebook, give yourself a high-five! Hopefully you should now have a strong foundation to build powerful, stateful experiences with tools like Code Interpreter, Retrieval, and Functions!

There's a few sections we didn't cover for the sake of brevity, so here's a few resources to explore further:

- [Annotations](https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages): parsing file citations
- [Files](https://platform.openai.com/docs/api-reference/assistants/file-object): Thread scoped vs Assistant scoped
- [Parallel Function Calls](https://platform.openai.com/docs/guides/function-calling/parallel-function-calling): calling multiple tools in a single Step
- Multi-Assistant Thread Runs: single Thread with Messages from multiple Assistants
- Streaming: coming soon!

Now go off and build something ama[zing](https://www.youtube.com/watch?v=xvFZjo5PgG0&pp=ygUQcmljayByb2xsIG5vIGFkcw%3D%3D)!


key: _node_content
string_value:
{"id_": "0e2a91a3-5b88-498d-8dda-5e8d3d67a553", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "254d9ba9-93cd-4373-ae1f-e51f3adf722f", "node_type": "1", "metadata": {}, "hash": "d28629c8f0ef1c8c0d397c8dea2723cc8c7ff5737f4c18aef09a5fccc70ad13e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Multimodal RAG with CLIP Embeddings and GPT-4 Vision


Multimodal RAG integrates additional modalities into traditional text-based RAG, enhancing LLMs' question-answering by providing extra context and grounding textual data for improved understanding.

Adopting the approach from the [clothing matchmaker cookbook](https://cookbook.openai.com/examples/how_to_combine_gpt4v_with_rag_outfit_assistant), we directly embed images for similarity search, bypassing the lossy process of text captioning, to boost retrieval accuracy.

Using CLIP-based embeddings further allows fine-tuning with specific data or updating with unseen images.

This technique is showcased through searching an enterprise knowledge base with user-provided tech images to deliver pertinent information.

# Installations

First let's install the relevant packages.


```python
#installations
%pip install clip
%pip install torch
%pip install pillow
%pip install faiss-cpu
%pip install numpy
%pip install git+https://github.com/openai/CLIP.git
%pip install openai
```

Then let's import all the needed packages.



```python
# model imports
import faiss
import json
import torch
from openai import OpenAI
import torch.nn as nn
from torch.utils.data import DataLoader
import clip
client = OpenAI()

# helper imports
from tqdm import tqdm
import json
import os
import numpy as np
import pickle
from typing import List, Union, Tuple

# visualisation imports
from PIL import Image
import matplotlib.pyplot as plt
import base64
```

Now let's load the CLIP model.


```python
#load model on device. The device you are running inference/training on is either a CPU or GPU if you have.
device = "cpu"
model, preprocess = clip.load("ViT-B/32",device=device)
```


We will now:
1.   Create the image embedding database
2.   Set up a query to the vision model
3.   Perform the semantic search
4.   Pass a user query to the image



# Create image embedding database

Next we will create our image embeddings knowledge base from a directory of images. This will be the knowledge base of technology that we search through to provide information to the user for an image they upload.

We pass in the directory in which we store our images (as JPEGs) and loop through each to create our embeddings.

We also have a description.json. This has an entry for every single image in our knowledge base. It has two keys: 'image_path' and 'description'. It maps each image to a useful description of this image to aid in answering the user question.

First let's write a function to get all the image paths in a given directory. We will then get all the jpeg's from a directory called 'image_database'


key: _node_content
string_value:
{"id_": "254d9ba9-93cd-4373-ae1f-e51f3adf722f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e2a91a3-5b88-498d-8dda-5e8d3d67a553", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5b438987c97c0dc300c86b5215050a621a121760dd14b1753dff91aa11b9f2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39e7afac-d610-4430-9777-8856620f9353", "node_type": "1", "metadata": {}, "hash": "df9406da5b60ee611b5e904582a3d3c7a8aa91d63fa509c8a49cfd04a0db5441", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1087, "end_char_idx": 4390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# model imports
import faiss
import json
import torch
from openai import OpenAI
import torch.nn as nn
from torch.utils.data import DataLoader
import clip
client = OpenAI()

# helper imports
from tqdm import tqdm
import json
import os
import numpy as np
import pickle
from typing import List, Union, Tuple

# visualisation imports
from PIL import Image
import matplotlib.pyplot as plt
import base64
```

Now let's load the CLIP model.


```python
#load model on device. The device you are running inference/training on is either a CPU or GPU if you have.
device = "cpu"
model, preprocess = clip.load("ViT-B/32",device=device)
```


We will now:
1.   Create the image embedding database
2.   Set up a query to the vision model
3.   Perform the semantic search
4.   Pass a user query to the image



# Create image embedding database

Next we will create our image embeddings knowledge base from a directory of images. This will be the knowledge base of technology that we search through to provide information to the user for an image they upload.

We pass in the directory in which we store our images (as JPEGs) and loop through each to create our embeddings.

We also have a description.json. This has an entry for every single image in our knowledge base. It has two keys: 'image_path' and 'description'. It maps each image to a useful description of this image to aid in answering the user question.

First let's write a function to get all the image paths in a given directory. We will then get all the jpeg's from a directory called 'image_database'


```python
def get_image_paths(directory: str, number: int = None) -> List[str]:
    image_paths = []
    count = 0
    for filename in os.listdir(directory):
        if filename.endswith('.jpeg'):
            image_paths.append(os.path.join(directory, filename))
            if number is not None and count == number:
                return [image_paths[-1]]
            count += 1
    return image_paths
direc = 'image_database/'
image_paths = get_image_paths(direc)

```

Next we will write a function to get the image embeddings from the CLIP model given a series of paths.

We first preprocess the image using the preprocess function we got earlier. This performs a few things to ensure the input to the CLIP model is of the right format and dimensionality including resizing, normalization, colour channel adjustment etc.

We then stack these preprocessed images together so we can pass them into the model at once rather than in a loop. And finally return the model output which is an array of embeddings.


```python
def get_features_from_image_path(image_paths):
  images = [preprocess(Image.open(image_path).convert("RGB")) for image_path in image_paths]
  image_input = torch.tensor(np.stack(images))
  with torch.no_grad():
    image_features = model.encode_image(image_input).float()
  return image_features
image_features = get_features_from_image_path(image_paths)

```

We can now create our vector database.


```python
index = faiss.IndexFlatIP(image_features.shape[1])
index.add(image_features)

```

And also ingest our json for image-description mapping and create a list of jsons. We also create a helper function to search through this list for a given image we want, so we can obtain the description of that image


key: _node_content
string_value:
{"id_": "39e7afac-d610-4430-9777-8856620f9353", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "254d9ba9-93cd-4373-ae1f-e51f3adf722f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1e193681596d47fa73315b27570b46ad0aeb2daa425fa9c492e132198559ef8f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d8bdefc3-6669-44b3-8951-0289b2779298", "node_type": "1", "metadata": {}, "hash": "150a95265ed3de04c6a2404ad2962375163fd558b4958201bc802d6ed6713ccc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2654, "end_char_idx": 5348, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_image_paths(directory: str, number: int = None) -> List[str]:
    image_paths = []
    count = 0
    for filename in os.listdir(directory):
        if filename.endswith('.jpeg'):
            image_paths.append(os.path.join(directory, filename))
            if number is not None and count == number:
                return [image_paths[-1]]
            count += 1
    return image_paths
direc = 'image_database/'
image_paths = get_image_paths(direc)

```

Next we will write a function to get the image embeddings from the CLIP model given a series of paths.

We first preprocess the image using the preprocess function we got earlier. This performs a few things to ensure the input to the CLIP model is of the right format and dimensionality including resizing, normalization, colour channel adjustment etc.

We then stack these preprocessed images together so we can pass them into the model at once rather than in a loop. And finally return the model output which is an array of embeddings.


```python
def get_features_from_image_path(image_paths):
  images = [preprocess(Image.open(image_path).convert("RGB")) for image_path in image_paths]
  image_input = torch.tensor(np.stack(images))
  with torch.no_grad():
    image_features = model.encode_image(image_input).float()
  return image_features
image_features = get_features_from_image_path(image_paths)

```

We can now create our vector database.


```python
index = faiss.IndexFlatIP(image_features.shape[1])
index.add(image_features)

```

And also ingest our json for image-description mapping and create a list of jsons. We also create a helper function to search through this list for a given image we want, so we can obtain the description of that image


```python
data = []
image_path = 'train1.jpeg'
with open('description.json', 'r') as file:
    for line in file:
        data.append(json.loads(line))
def find_entry(data, key, value):
    for entry in data:
        if entry.get(key) == value:
            return entry
    return None
```

Let us display an example image, this will be the user uploaded image. This is a piece of tech that was unveiled at the 2024 CES. It is the DELTA Pro Ultra Whole House Battery Generator.


```python
im = Image.open(image_path)
plt.imshow(im)
plt.show()
```

![Delta Pro](../images/train1.jpeg)

# Querying the vision model

Now let's have a look at what GPT-4 Vision (which wouldn't have seen this technology before) will label it as.



First we will need to write a function to encode our image in base64 as this is the format we will pass into the vision model. Then we will create a generic image_query function to allow us to query the LLM with an image input.


key: _node_content
string_value:
{"id_": "d8bdefc3-6669-44b3-8951-0289b2779298", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39e7afac-d610-4430-9777-8856620f9353", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1b4493a1434efb914eb3a1a02f552fb15ac9dc65a67449eb66c821305a4cbb77", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc920f39-07a8-43cf-b118-676551c4074e", "node_type": "1", "metadata": {}, "hash": "100df2a64d14e9d8acc948172ca42df4e10a6639250ac3f880dec1bd2c15c032", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3668, "end_char_idx": 7007, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_features_from_image_path(image_paths):
  images = [preprocess(Image.open(image_path).convert("RGB")) for image_path in image_paths]
  image_input = torch.tensor(np.stack(images))
  with torch.no_grad():
    image_features = model.encode_image(image_input).float()
  return image_features
image_features = get_features_from_image_path(image_paths)

```

We can now create our vector database.


```python
index = faiss.IndexFlatIP(image_features.shape[1])
index.add(image_features)

```

And also ingest our json for image-description mapping and create a list of jsons. We also create a helper function to search through this list for a given image we want, so we can obtain the description of that image


```python
data = []
image_path = 'train1.jpeg'
with open('description.json', 'r') as file:
    for line in file:
        data.append(json.loads(line))
def find_entry(data, key, value):
    for entry in data:
        if entry.get(key) == value:
            return entry
    return None
```

Let us display an example image, this will be the user uploaded image. This is a piece of tech that was unveiled at the 2024 CES. It is the DELTA Pro Ultra Whole House Battery Generator.


```python
im = Image.open(image_path)
plt.imshow(im)
plt.show()
```

![Delta Pro](../images/train1.jpeg)

# Querying the vision model

Now let's have a look at what GPT-4 Vision (which wouldn't have seen this technology before) will label it as.



First we will need to write a function to encode our image in base64 as this is the format we will pass into the vision model. Then we will create a generic image_query function to allow us to query the LLM with an image input.


```python
def encode_image(image_path):
    with open(image_path, 'rb') as image_file:
        encoded_image = base64.b64encode(image_file.read())
        return encoded_image.decode('utf-8')

def image_query(query, image_path):
    response = client.chat.completions.create(
        model='gpt-4-vision-preview',
        messages=[
            {
            "role": "user",
            "content": [
                {
                "type": "text",
                "text": query,
                },
                {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{encode_image(image_path)}",
                },
                }
            ],
            }
        ],
        max_tokens=300,
    )
    # Extract relevant features from the response
    return response.choices[0].message.content
image_query('Write a short label of what is show in this image?', image_path)
```




    'Autonomous Delivery Robot'



As we can see, it tries its best from the information it's been trained on but it makes a mistake due to it not having seen anything similar in its training data. This is because it is an ambiguous image making it difficult to extrapolate and deduce.

# Performing semantic search

Now let's perform similarity search to find the two most similar images in our knowledge base. We do this by getting the embeddings of a user inputted image_path, retrieving the indexes and distances of the similar iamges in our database. Distance will be our proxy metric for similarity and a smaller distance means more similar. We then sort based on distance in descending order.


key: _node_content
string_value:
{"id_": "dc920f39-07a8-43cf-b118-676551c4074e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d8bdefc3-6669-44b3-8951-0289b2779298", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5f18632c7989875e34d7006e972d63d8d4174442540f821feaf0f378bc67fa46", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bd2a2b1-2032-46f5-85cd-d3757b116b03", "node_type": "1", "metadata": {}, "hash": "33c1a918809db5d71f4d35398935deca9e416e03cf21e4b98bb757e7b0795d95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5121, "end_char_idx": 8620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
First we will need to write a function to encode our image in base64 as this is the format we will pass into the vision model. Then we will create a generic image_query function to allow us to query the LLM with an image input.


```python
def encode_image(image_path):
    with open(image_path, 'rb') as image_file:
        encoded_image = base64.b64encode(image_file.read())
        return encoded_image.decode('utf-8')

def image_query(query, image_path):
    response = client.chat.completions.create(
        model='gpt-4-vision-preview',
        messages=[
            {
            "role": "user",
            "content": [
                {
                "type": "text",
                "text": query,
                },
                {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{encode_image(image_path)}",
                },
                }
            ],
            }
        ],
        max_tokens=300,
    )
    # Extract relevant features from the response
    return response.choices[0].message.content
image_query('Write a short label of what is show in this image?', image_path)
```




    'Autonomous Delivery Robot'



As we can see, it tries its best from the information it's been trained on but it makes a mistake due to it not having seen anything similar in its training data. This is because it is an ambiguous image making it difficult to extrapolate and deduce.

# Performing semantic search

Now let's perform similarity search to find the two most similar images in our knowledge base. We do this by getting the embeddings of a user inputted image_path, retrieving the indexes and distances of the similar iamges in our database. Distance will be our proxy metric for similarity and a smaller distance means more similar. We then sort based on distance in descending order.


```python
image_search_embedding = get_features_from_image_path([image_path])
distances, indices = index.search(image_search_embedding.reshape(1, -1), 2) #2 signifies the number of topmost similar images to bring back
distances = distances[0]
indices = indices[0]
indices_distances = list(zip(indices, distances))
indices_distances.sort(key=lambda x: x[1], reverse=True)
```

We require the indices as we will use this to serach through our image_directory and selecting the image at the location of the index to feed into the vision model for RAG.

And let's see what it brought back (we display these in order of similarity):


```python
#display similar images
for idx, distance in indices_distances:
    print(idx)
    path = get_image_paths(direc, idx)[0]
    im = Image.open(path)
    plt.imshow(im)
    plt.show()
```

![Delta Pro2](../images/train2.jpeg)

![Delta Pro3](../images/train17.jpeg)

We can see here it brought back two images which contain the DELTA Pro Ultra Whole House Battery Generator. In one of the images it also has some background which could be distracting but manages to find the right image.

# User querying the most similar image

Now for our most similar image, we want to pass it and the description of it to gpt-v with a user query so they can inquire about the technology that they may have bought. This is where the power of the vision model comes in, where you can ask general queries for which the model hasn't been explicitly trained on to the model and it responds with high accuracy.

In our example below, we will inquire as to the capacity of the item in question.


key: _node_content
string_value:
{"id_": "4bd2a2b1-2032-46f5-85cd-d3757b116b03", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25ff69a0-40fb-45c5-abc0-c787369edbdc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3543bca4cc5dc3a9c45cff88256635f96e56b47cdca9cfdfa6b0f2f33d5e7cc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc920f39-07a8-43cf-b118-676551c4074e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/custom_image_embedding_search.ipynb", "title": "Multimodal RAG with CLIP Embeddings and GPT-4 Vision", "tokens": 2176, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fafa27fcb899d6fbb21954f5edfbaf7e7946f1baa3034872a3fd8f97477c3112", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7010, "end_char_idx": 9951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
image_search_embedding = get_features_from_image_path([image_path])
distances, indices = index.search(image_search_embedding.reshape(1, -1), 2) #2 signifies the number of topmost similar images to bring back
distances = distances[0]
indices = indices[0]
indices_distances = list(zip(indices, distances))
indices_distances.sort(key=lambda x: x[1], reverse=True)
```

We require the indices as we will use this to serach through our image_directory and selecting the image at the location of the index to feed into the vision model for RAG.

And let's see what it brought back (we display these in order of similarity):


```python
#display similar images
for idx, distance in indices_distances:
    print(idx)
    path = get_image_paths(direc, idx)[0]
    im = Image.open(path)
    plt.imshow(im)
    plt.show()
```

![Delta Pro2](../images/train2.jpeg)

![Delta Pro3](../images/train17.jpeg)

We can see here it brought back two images which contain the DELTA Pro Ultra Whole House Battery Generator. In one of the images it also has some background which could be distracting but manages to find the right image.

# User querying the most similar image

Now for our most similar image, we want to pass it and the description of it to gpt-v with a user query so they can inquire about the technology that they may have bought. This is where the power of the vision model comes in, where you can ask general queries for which the model hasn't been explicitly trained on to the model and it responds with high accuracy.

In our example below, we will inquire as to the capacity of the item in question.


```python
similar_path = get_image_paths(direc, indices_distances[0][0])[0]
element = find_entry(data, 'image_path', similar_path)

user_query = 'What is the capacity of this item?'
prompt = f"""
Below is a user query, I want you to answer the query using the description and image provided.

user query:
{user_query}

description:
{element['description']}
"""
image_query(prompt, similar_path)
```




    'The portable home battery DELTA Pro has a base capacity of 3.6kWh. This capacity can be expanded up to 25kWh with additional batteries. The image showcases the DELTA Pro, which has an impressive 3600W power capacity for AC output as well.'



And we see it is able to answer the question. This was only possible by matching images directly and from there gathering the relevant description as context.

# Conclusion

In this notebook, we have gone through how to use the CLIP model, an example of creating an image embedding database using the CLIP model, performing semantic search and finally providing a user query to answer the question.

The applications of this pattern of usage spread across many different application domains and this is easily improved to further enhance the technique. For example you may finetune CLIP, you may improve the retrieval process just like in RAG and you can prompt engineer GPT-V.


key: _node_content
string_value:
{"id_": "66f38ea7-e8c0-45b7-8c9d-ef544cff03ab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f756d9b6-414b-45a3-a8e5-4ee6ff5b97c2", "node_type": "1", "metadata": {}, "hash": "6dfb980ced0bdc8480693e2bd14274e0c27a8c263037566190e3b1bfa7b12d00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using GPT-4o mini to tag & caption images

This notebook explores how to leverage the vision capabilities of the GPT-4* models (for example `gpt-4o`, `gpt-4o-mini` or `gpt-4-turbo`) to tag & caption images. 

We can leverage the multimodal capabilities of these models to provide input images along with additional context on what they represent, and prompt the model to output tags or image descriptions. The image descriptions can then be further refined with a language model (in this notebook, we'll use `gpt-4o-mini`) to generate captions. 

Generating text content from images can be useful for multiple use cases, especially use cases involving search.  
We will illustrate a search use case in this notebook by using generated keywords and product captions to search for products - both from a text input and an image input.

As an example, we will use a dataset of Amazon furniture items, tag them with relevant keywords and generate short, descriptive captions.

## Setup


```python
# Install dependencies if needed
%pip install openai
%pip install scikit-learn
```


```python
from IPython.display import Image, display
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from openai import OpenAI

# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python
client = OpenAI()
```


```python
# Loading dataset
dataset_path =  "data/amazon_furniture_dataset.csv"
df = pd.read_csv(dataset_path)
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>asin</th>
      <th>url</th>
      <th>title</th>
      <th>brand</th>
      <th>price</th>
      <th>availability</th>
      <th>categories</th>
      <th>primary_image</th>
      <th>images</th>
      <th>upc</th>
      <th>.</th>
      <th>color</th>
      <th>material</th>
      <th>style</th>
      <th>important_information</th>
      <th>product_overview</th>
      <th>about_item</th>
      <th>description</th>
      <th>specifications</th>
      <th>uniq_id</th>
      <th>scraped_at</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B0CJHKVG6P</td>
      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>
      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye.</td>
      <td>GOYMFK</td>
      <td>$24.99</td>
      <td>Only 13 left in stock - order soon.</td>
      <td>['Home &amp; Kitchen', 'Storage &amp; Organization', '.</td>
      <td>https://m.media-amazon.


key: _node_content
string_value:
{"id_": "f756d9b6-414b-45a3-a8e5-4ee6ff5b97c2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66f38ea7-e8c0-45b7-8c9d-ef544cff03ab", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "bd13a6b7efae9b583728c9bf1d74a8af631ced768be1a187e58d5f77b951248c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3483245c-dbbb-4234-a939-2ba60782e810", "node_type": "1", "metadata": {}, "hash": "4c1f77a34804d664a733d1047a5c1da8edfed8a2349687b5627ee9b0a9447706", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>asin</th>
      <th>url</th>
      <th>title</th>
      <th>brand</th>
      <th>price</th>
      <th>availability</th>
      <th>categories</th>
      <th>primary_image</th>
      <th>images</th>
      <th>upc</th>
      <th>.</th>
      <th>color</th>
      <th>material</th>
      <th>style</th>
      <th>important_information</th>
      <th>product_overview</th>
      <th>about_item</th>
      <th>description</th>
      <th>specifications</th>
      <th>uniq_id</th>
      <th>scraped_at</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>B0CJHKVG6P</td>
      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>
      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye.</td>
      <td>GOYMFK</td>
      <td>$24.99</td>
      <td>Only 13 left in stock - order soon.</td>
      <td>['Home &amp; Kitchen', 'Storage &amp; Organization', '.</td>
      <td>https://m.media-amazon.com/images/I/416WaLx10j.</td>
      <td>['https://m.media-amazon.com/images/I/416WaLx1.</td>
      <td>NaN</td>
      <td>.</td>
      <td>White</td>
      <td>Metal</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' GOYMFK '}, {'Color': ' White '}, .</td>
      <td>['Multiple layers: Provides ample storage spac.</td>
      <td>multiple shoes, coats, hats, and other items E.</td>
      <td>['Brand: GOYMFK', 'Color: White', 'Material: M.</td>
      <td>02593e81-5c09-5069-8516-b0b29f439ded</td>
      <td>2024-02-02 15:15:08</td>
    </tr>
    <tr>
      <th>1</th>
      <td>B0B66QHB23</td>
      <td>https://www.amazon.com/dp/B0B66QHB23</td>
      <td>subrtex Leather ding Room, Dining Chairs Set o.</td>
      <td>subrtex</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>['Home &amp; Kitchen', 'Furniture', 'Dining Room F.</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7.</td>
      <td>['https://m.


key: _node_content
string_value:
{"id_": "3483245c-dbbb-4234-a939-2ba60782e810", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f756d9b6-414b-45a3-a8e5-4ee6ff5b97c2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cc0288d6d7202dafde0accbe82391932be9ba7a8afb44237f9b89762bdc2c714", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ab93145-1f7f-4b9c-a9fd-b09b990a0642", "node_type": "1", "metadata": {}, "hash": "7f60dfcf585843fd2b85a2c30603ca8a71d70db1ba8fbddd13514b52089bf4df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Kitchen', 'Storage &amp; Organization', '.</td>
      <td>https://m.media-amazon.com/images/I/416WaLx10j.</td>
      <td>['https://m.media-amazon.com/images/I/416WaLx1.</td>
      <td>NaN</td>
      <td>.</td>
      <td>White</td>
      <td>Metal</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' GOYMFK '}, {'Color': ' White '}, .</td>
      <td>['Multiple layers: Provides ample storage spac.</td>
      <td>multiple shoes, coats, hats, and other items E.</td>
      <td>['Brand: GOYMFK', 'Color: White', 'Material: M.</td>
      <td>02593e81-5c09-5069-8516-b0b29f439ded</td>
      <td>2024-02-02 15:15:08</td>
    </tr>
    <tr>
      <th>1</th>
      <td>B0B66QHB23</td>
      <td>https://www.amazon.com/dp/B0B66QHB23</td>
      <td>subrtex Leather ding Room, Dining Chairs Set o.</td>
      <td>subrtex</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>['Home &amp; Kitchen', 'Furniture', 'Dining Room F.</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7.</td>
      <td>['https://m.media-amazon.com/images/I/31SejUEW.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Black</td>
      <td>Sponge</td>
      <td>Black Rubber Wood</td>
      <td>[]</td>
      <td>NaN</td>
      <td>['【Easy Assembly】: Set of 2 dining room chairs.</td>
      <td>subrtex Dining chairs Set of 2</td>
      <td>['Brand: subrtex', 'Color: Black', 'Product Di.</td>
      <td>5938d217-b8c5-5d3e-b1cf-e28e340f292e</td>
      <td>2024-02-02 15:15:09</td>
    </tr>
    <tr>
      <th>2</th>
      <td>B0BXRTWLYK</td>
      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>
      <td>Plant Repotting Mat MUYETOL Waterproof Transpl.</td>
      <td>MUYETOL</td>
      <td>$5.98</td>
      <td>In Stock</td>
      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo.</td>
      <td>https://m.media-amazon.com/images/I/41RgefVq70.</td>
      <td>['https://m.media-amazon.com/images/I/41RgefVq.</td>
      <td>NaN</td>
      <td>.


key: _node_content
string_value:
{"id_": "9ab93145-1f7f-4b9c-a9fd-b09b990a0642", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3483245c-dbbb-4234-a939-2ba60782e810", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "63d4c6b029b91a9e8268bb67988760bf4516977e2833eff8abc73af2703bbffc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dd2dade-9db0-4275-a3b7-54f5f27cdea2", "node_type": "1", "metadata": {}, "hash": "3a5c3a2f6162f4c42d51aafdf38d36e4ff0ab0c89ae26f0e9ffc9ceaf34faf87", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
com/images/I/31SejUEWY7.</td>
      <td>['https://m.media-amazon.com/images/I/31SejUEW.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Black</td>
      <td>Sponge</td>
      <td>Black Rubber Wood</td>
      <td>[]</td>
      <td>NaN</td>
      <td>['【Easy Assembly】: Set of 2 dining room chairs.</td>
      <td>subrtex Dining chairs Set of 2</td>
      <td>['Brand: subrtex', 'Color: Black', 'Product Di.</td>
      <td>5938d217-b8c5-5d3e-b1cf-e28e340f292e</td>
      <td>2024-02-02 15:15:09</td>
    </tr>
    <tr>
      <th>2</th>
      <td>B0BXRTWLYK</td>
      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>
      <td>Plant Repotting Mat MUYETOL Waterproof Transpl.</td>
      <td>MUYETOL</td>
      <td>$5.98</td>
      <td>In Stock</td>
      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo.</td>
      <td>https://m.media-amazon.com/images/I/41RgefVq70.</td>
      <td>['https://m.media-amazon.com/images/I/41RgefVq.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Green</td>
      <td>Polyethylene</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' MUYETOL '}, {'Size': ' 26.8*26.8 .</td>
      <td>['PLANT REPOTTING MAT SIZE: 26.8" x 26.8", squ.</td>
      <td>NaN</td>
      <td>['Brand: MUYETOL', 'Size: 26.8*26.8', 'Item We.</td>
      <td>b2ede786-3f51-5a45-9a5b-bcf856958cd8</td>
      <td>2024-02-02 15:15:09</td>
    </tr>
    <tr>
      <th>3</th>
      <td>B0C1MRB2M8</td>
      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>
      <td>Pickleball Doormat, Welcome Doormat Absorbent .</td>
      <td>VEWETOL</td>
      <td>$13.99</td>
      <td>Only 10 left in stock - order soon.</td>
      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo.</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler.</td>
      <td>['https://m.media-amazon.com/images/I/61vz1Igl.


key: _node_content
string_value:
{"id_": "8dd2dade-9db0-4275-a3b7-54f5f27cdea2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ab93145-1f7f-4b9c-a9fd-b09b990a0642", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d533c926cee854fbf7a36b72a9c78db258a16dd20859a58cf392b9adab439489", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0646027-5dcb-4e5d-a218-15a435ffd780", "node_type": "1", "metadata": {}, "hash": "74c97883a7360fef58164778be34cb7fc9a930b1da80ef1b9dc2ff1315e51a5a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
com/images/I/41RgefVq.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Green</td>
      <td>Polyethylene</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' MUYETOL '}, {'Size': ' 26.8*26.8 .</td>
      <td>['PLANT REPOTTING MAT SIZE: 26.8" x 26.8", squ.</td>
      <td>NaN</td>
      <td>['Brand: MUYETOL', 'Size: 26.8*26.8', 'Item We.</td>
      <td>b2ede786-3f51-5a45-9a5b-bcf856958cd8</td>
      <td>2024-02-02 15:15:09</td>
    </tr>
    <tr>
      <th>3</th>
      <td>B0C1MRB2M8</td>
      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>
      <td>Pickleball Doormat, Welcome Doormat Absorbent .</td>
      <td>VEWETOL</td>
      <td>$13.99</td>
      <td>Only 10 left in stock - order soon.</td>
      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo.</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler.</td>
      <td>['https://m.media-amazon.com/images/I/61vz1Igl.</td>
      <td>NaN</td>
      <td>.</td>
      <td>A5589</td>
      <td>Rubber</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' VEWETOL '}, {'Size': ' 16*24INCH .</td>
      <td>['Specifications: 16x24 Inch ', " High-Quality.</td>
      <td>The decorative doormat features a subtle textu.</td>
      <td>['Brand: VEWETOL', 'Size: 16*24INCH', 'Materia.</td>
      <td>8fd9377b-cfa6-5f10-835c-6b8eca2816b5</td>
      <td>2024-02-02 15:15:10</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B0CG1N9QRC</td>
      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>
      <td>JOIN IRON Foldable TV Trays for Eating Set of .</td>
      <td>JOIN IRON Store</td>
      <td>$89.99</td>
      <td>Usually ships within 5 to 6 weeks</td>
      <td>['Home &amp; Kitchen', 'Furniture', 'Game &amp; Recrea.</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN.</td>
      <td>['https://m.


key: _node_content
string_value:
{"id_": "e0646027-5dcb-4e5d-a218-15a435ffd780", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dd2dade-9db0-4275-a3b7-54f5f27cdea2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "edfb393c4af0f1dc019a2e63aae610b1ed7db5f0eb668ab248d7e6570a816b78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "994610d5-211b-40a1-ae53-9199323a8c63", "node_type": "1", "metadata": {}, "hash": "28fa1c9d5b6be5dce72abf1315c58a3a0712626db27a519c717015077f2ae9c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>['https://m.media-amazon.com/images/I/61vz1Igl.</td>
      <td>NaN</td>
      <td>.</td>
      <td>A5589</td>
      <td>Rubber</td>
      <td>Modern</td>
      <td>[]</td>
      <td>[{'Brand': ' VEWETOL '}, {'Size': ' 16*24INCH .</td>
      <td>['Specifications: 16x24 Inch ', " High-Quality.</td>
      <td>The decorative doormat features a subtle textu.</td>
      <td>['Brand: VEWETOL', 'Size: 16*24INCH', 'Materia.</td>
      <td>8fd9377b-cfa6-5f10-835c-6b8eca2816b5</td>
      <td>2024-02-02 15:15:10</td>
    </tr>
    <tr>
      <th>4</th>
      <td>B0CG1N9QRC</td>
      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>
      <td>JOIN IRON Foldable TV Trays for Eating Set of .</td>
      <td>JOIN IRON Store</td>
      <td>$89.99</td>
      <td>Usually ships within 5 to 6 weeks</td>
      <td>['Home &amp; Kitchen', 'Furniture', 'Game &amp; Recrea.</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN.</td>
      <td>['https://m.media-amazon.com/images/I/41p4d4VJ.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Grey Set of 4</td>
      <td>Iron</td>
      <td>X Classic Style</td>
      <td>[]</td>
      <td>NaN</td>
      <td>['Includes 4 Folding Tv Tray Tables And one Co.</td>
      <td>Set of Four Folding Trays With Matching Storag.</td>
      <td>['Brand: JOIN IRON', 'Shape: Rectangular', 'In.</td>
      <td>bdc9aa30-9439-50dc-8e89-213ea211d66a</td>
      <td>2024-02-02 15:15:11</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 25 columns</p>
</div>



## Tag images

In this section, we'll use GPT-4o mini to generate relevant tags for our products.

We'll use a simple zero-shot approach to extract keywords, and deduplicate those keywords using embeddings to avoid having multiple keywords that are too similar.

We will use a combination of an image and the product title to avoid extracting keywords for other items that are depicted in the image - sometimes there are multiple items used in the scene and we want to focus on just the one we want to tag.

### Extract keywords


key: _node_content
string_value:
{"id_": "994610d5-211b-40a1-ae53-9199323a8c63", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0646027-5dcb-4e5d-a218-15a435ffd780", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1433adbee6696ebe5502e55710abd2489d213b573247ead8f28c001c05c82f30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abe7654f-5898-4293-ae1c-380a74d58f68", "node_type": "1", "metadata": {}, "hash": "59c03e3a90d1823ab6f448bbe8f2796ef220a386a71ee6b593ca058d9c15cfd6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Kitchen', 'Furniture', 'Game &amp; Recrea.</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN.</td>
      <td>['https://m.media-amazon.com/images/I/41p4d4VJ.</td>
      <td>NaN</td>
      <td>.</td>
      <td>Grey Set of 4</td>
      <td>Iron</td>
      <td>X Classic Style</td>
      <td>[]</td>
      <td>NaN</td>
      <td>['Includes 4 Folding Tv Tray Tables And one Co.</td>
      <td>Set of Four Folding Trays With Matching Storag.</td>
      <td>['Brand: JOIN IRON', 'Shape: Rectangular', 'In.</td>
      <td>bdc9aa30-9439-50dc-8e89-213ea211d66a</td>
      <td>2024-02-02 15:15:11</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 25 columns</p>
</div>



## Tag images

In this section, we'll use GPT-4o mini to generate relevant tags for our products.

We'll use a simple zero-shot approach to extract keywords, and deduplicate those keywords using embeddings to avoid having multiple keywords that are too similar.

We will use a combination of an image and the product title to avoid extracting keywords for other items that are depicted in the image - sometimes there are multiple items used in the scene and we want to focus on just the one we want to tag.

### Extract keywords


```python
system_prompt = '''
    You are an agent specialized in tagging images of furniture items, decorative items, or furnishings with relevant keywords that could be used to search for these items on a marketplace.
    
    You will be provided with an image and the title of the item that is depicted in the image, and your goal is to extract keywords for only the item specified. 
    
    Keywords should be concise and in lower case. 
    
    Keywords can describe things like:
    - Item type e.g. 'sofa bed', 'chair', 'desk', 'plant'
    - Item material e.g. 'wood', 'metal', 'fabric'
    - Item style e.g. 'scandinavian', 'vintage', 'industrial'
    - Item color e.g. 'red', 'blue', 'white'
    
    Only deduce material, style or color keywords when it is obvious that they make the item depicted in the image stand out.

    Return keywords in the format of an array of strings, like this:
    ['desk', 'industrial', 'metal']
    
'''

def analyze_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
        max_tokens=300,
        top_p=0.1
    )

    return response.choices[0].message.content
```

#### Testing with a few examples


```python
examples = df.iloc[:5]
```


key: _node_content
string_value:
{"id_": "abe7654f-5898-4293-ae1c-380a74d58f68", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "994610d5-211b-40a1-ae53-9199323a8c63", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2acc0c90da4a8e021a83267c930c13b96991dbe41410f4f87c91444dc4f9a15b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82a1d505-e6a4-4aac-9595-e0549351b5b3", "node_type": "1", "metadata": {}, "hash": "3fd45ea04c91920887abd5f07c6997cb68a6e99cfd3893d3672a82a7715b5ec6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7501, "end_char_idx": 10280, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
system_prompt = '''
    You are an agent specialized in tagging images of furniture items, decorative items, or furnishings with relevant keywords that could be used to search for these items on a marketplace.
    
    You will be provided with an image and the title of the item that is depicted in the image, and your goal is to extract keywords for only the item specified. 
    
    Keywords should be concise and in lower case. 
    
    Keywords can describe things like:
    - Item type e.g. 'sofa bed', 'chair', 'desk', 'plant'
    - Item material e.g. 'wood', 'metal', 'fabric'
    - Item style e.g. 'scandinavian', 'vintage', 'industrial'
    - Item color e.g. 'red', 'blue', 'white'
    
    Only deduce material, style or color keywords when it is obvious that they make the item depicted in the image stand out.

    Return keywords in the format of an array of strings, like this:
    ['desk', 'industrial', 'metal']
    
'''

def analyze_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
        max_tokens=300,
        top_p=0.1
    )

    return response.choices[0].message.content
```

#### Testing with a few examples


```python
examples = df.iloc[:5]
```


```python
for index, ex in examples.iterrows():
    url = ex['primary_image']
    img = Image(url=url)
    display(img)
    result = analyze_image(url, ex['title'])
    print(result)
    print("\n\n")
```


<img src="https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg"/>


    ['shoe rack', 'metal', 'white', 'multi-layer', 'hooks']
    
    
    



<img src="https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg"/>


    ['dining chair', 'leather', 'black']
    
    
    



<img src="https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg"/>


    ['repotting mat', 'waterproof', 'portable', 'foldable', 'green']
    
    
    



<img src="https://m.media-amazon.com/images/I/61vz1IglerL._SS522_.jpg"/>


    ['doormat', 'absorbent', 'non-slip', 'coconut fiber', 'welcome', 'pickleball', 'outdoor']
    
    
    



<img src="https://m.media-amazon.com/images/I/41p4d4VJnNL._SS522_.jpg"/>


    ['tv tray', 'foldable', 'metal', 'grey']
    
    
    


### Looking up existing keywords

Using embeddings to avoid duplicates (synonyms) and/or match pre-defined keywords


key: _node_content
string_value:
{"id_": "82a1d505-e6a4-4aac-9595-e0549351b5b3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abe7654f-5898-4293-ae1c-380a74d58f68", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "31112d2459eb7d5594f6137996a90bb6b8405b2ebba8ff68916c07dab351d823", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8155ee64-3e62-40d7-8627-3973289e91ba", "node_type": "1", "metadata": {}, "hash": "1ccfb98d98f53a345666959c9cf610be75bdcb0aeaaa5d5df82010651e96f15b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9146, "end_char_idx": 10872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
examples = df.iloc[:5]
```


```python
for index, ex in examples.iterrows():
    url = ex['primary_image']
    img = Image(url=url)
    display(img)
    result = analyze_image(url, ex['title'])
    print(result)
    print("\n\n")
```


<img src="https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg"/>


    ['shoe rack', 'metal', 'white', 'multi-layer', 'hooks']
    
    
    



<img src="https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg"/>


    ['dining chair', 'leather', 'black']
    
    
    



<img src="https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg"/>


    ['repotting mat', 'waterproof', 'portable', 'foldable', 'green']
    
    
    



<img src="https://m.media-amazon.com/images/I/61vz1IglerL._SS522_.jpg"/>


    ['doormat', 'absorbent', 'non-slip', 'coconut fiber', 'welcome', 'pickleball', 'outdoor']
    
    
    



<img src="https://m.media-amazon.com/images/I/41p4d4VJnNL._SS522_.jpg"/>


    ['tv tray', 'foldable', 'metal', 'grey']
    
    
    


### Looking up existing keywords

Using embeddings to avoid duplicates (synonyms) and/or match pre-defined keywords


```python
# Feel free to change the embedding model here
def get_embedding(value, model="text-embedding-3-large"): 
    embeddings = client.embeddings.create(
      model=model,
      input=value,
      encoding_format="float"
    )
    return embeddings.data[0].embedding
```

#### Testing with example keywords


```python
# Existing keywords
keywords_list = ['industrial', 'metal', 'wood', 'vintage', 'bed']
```


```python
df_keywords = pd.DataFrame(keywords_list, columns=['keyword'])
df_keywords['embedding'] = df_keywords['keyword'].apply(lambda x: get_embedding(x))
df_keywords
```


key: _node_content
string_value:
{"id_": "8155ee64-3e62-40d7-8627-3973289e91ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82a1d505-e6a4-4aac-9595-e0549351b5b3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d79fea2cd67f2ac2e43b494f946467c6331b9d8f5c4331447c7711ab24b029d5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "920704d5-fa88-44bb-a9d3-dc36cd225199", "node_type": "1", "metadata": {}, "hash": "00353b2b9df603a3c213d0bcc20dcd65990ea3b8939474583c66a65c929ff324", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9471, "end_char_idx": 11930, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
['shoe rack', 'metal', 'white', 'multi-layer', 'hooks']
    
    
    



<img src="https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg"/>


    ['dining chair', 'leather', 'black']
    
    
    



<img src="https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg"/>


    ['repotting mat', 'waterproof', 'portable', 'foldable', 'green']
    
    
    



<img src="https://m.media-amazon.com/images/I/61vz1IglerL._SS522_.jpg"/>


    ['doormat', 'absorbent', 'non-slip', 'coconut fiber', 'welcome', 'pickleball', 'outdoor']
    
    
    



<img src="https://m.media-amazon.com/images/I/41p4d4VJnNL._SS522_.jpg"/>


    ['tv tray', 'foldable', 'metal', 'grey']
    
    
    


### Looking up existing keywords

Using embeddings to avoid duplicates (synonyms) and/or match pre-defined keywords


```python
# Feel free to change the embedding model here
def get_embedding(value, model="text-embedding-3-large"): 
    embeddings = client.embeddings.create(
      model=model,
      input=value,
      encoding_format="float"
    )
    return embeddings.data[0].embedding
```

#### Testing with example keywords


```python
# Existing keywords
keywords_list = ['industrial', 'metal', 'wood', 'vintage', 'bed']
```


```python
df_keywords = pd.DataFrame(keywords_list, columns=['keyword'])
df_keywords['embedding'] = df_keywords['keyword'].apply(lambda x: get_embedding(x))
df_keywords
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>keyword</th>
      <th>embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>industrial</td>
      <td>[-0.026137426, 0.021297162, -0.007273361, -0.0...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>metal</td>
      <td>[-0.020492474, 0.0044436487, -0.0110632675, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>wood</td>
      <td>[0.013840097, 0.029538965, 0.00064718135, -0.0...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>vintage</td>
      <td>[-0.052348174, 0.008181616, -0.015513194, 0.00...</td>
    </tr>
    <tr>
      <th>4</th>
      <td>bed</td>
      <td>[-0.011677503, 0.023275835, 0.0026937425, -0.0...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "920704d5-fa88-44bb-a9d3-dc36cd225199", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8155ee64-3e62-40d7-8627-3973289e91ba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2bfbfe8e4f81ca602f6be8c890a6855543aa0785719ad41e6d3bee2491e3865d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8aa53671-e7b1-4a68-bd91-3137292d6fd2", "node_type": "1", "metadata": {}, "hash": "4e32495f6951dbde0efd538ceea9d98a2c436c2a71d80861482f724b1cb02d8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11935, "end_char_idx": 13903, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def compare_keyword(keyword):
    embedded_value = get_embedding(keyword)
    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    most_similar = df_keywords.sort_values('similarity', ascending=False).iloc[0]
    return most_similar

def replace_keyword(keyword, threshold = 0.6):
    most_similar = compare_keyword(keyword)
    if most_similar['similarity'] > threshold:
        print(f"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'")
        return most_similar['keyword']
    return keyword
```


```python
# Example keywords to compare to our list of existing keywords
example_keywords = ['bed frame', 'wooden', 'vintage', 'old school', 'desk', 'table', 'old', 'metal', 'metallic', 'woody']
final_keywords = []

for k in example_keywords:
    final_keywords.append(replace_keyword(k))
    
final_keywords = set(final_keywords)
print(f"Final keywords: {final_keywords}")
```

    Replacing 'bed frame' with existing keyword: 'bed'
    Replacing 'wooden' with existing keyword: 'wood'
    Replacing 'vintage' with existing keyword: 'vintage'
    Replacing 'metal' with existing keyword: 'metal'
    Replacing 'metallic' with existing keyword: 'metal'
    Replacing 'woody' with existing keyword: 'wood'
    Final keywords: {'vintage', 'desk', 'wood', 'table', 'old', 'bed', 'metal', 'old school'}


## Generate captions

In this section, we'll use GPT-4o mini to generate an image description and then use a few-shot examples approach with GPT-4-turbo to generate captions from the images.

If few-shot examples are not enough for your use case, consider fine-tuning a model to get the generated captions to match the style & tone you are targeting. 


```python
# Cleaning up dataset columns
selected_columns = ['title', 'primary_image', 'style', 'material', 'color', 'url']
df = df[selected_columns].copy()
df.head()
```


key: _node_content
string_value:
{"id_": "8aa53671-e7b1-4a68-bd91-3137292d6fd2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "920704d5-fa88-44bb-a9d3-dc36cd225199", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ce8929a84566c2a56e6765ae1abe24b66d97d7f7dcc47eea9ab877e0804907c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b67b02cc-60fe-4ba8-a2e7-5b6b833fa0b8", "node_type": "1", "metadata": {}, "hash": "ab8f0be8c62138c2ba86e8e8230829e26ef08e354998f8c59879248fceb54507", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12579, "end_char_idx": 15880, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Example keywords to compare to our list of existing keywords
example_keywords = ['bed frame', 'wooden', 'vintage', 'old school', 'desk', 'table', 'old', 'metal', 'metallic', 'woody']
final_keywords = []

for k in example_keywords:
    final_keywords.append(replace_keyword(k))
    
final_keywords = set(final_keywords)
print(f"Final keywords: {final_keywords}")
```

    Replacing 'bed frame' with existing keyword: 'bed'
    Replacing 'wooden' with existing keyword: 'wood'
    Replacing 'vintage' with existing keyword: 'vintage'
    Replacing 'metal' with existing keyword: 'metal'
    Replacing 'metallic' with existing keyword: 'metal'
    Replacing 'woody' with existing keyword: 'wood'
    Final keywords: {'vintage', 'desk', 'wood', 'table', 'old', 'bed', 'metal', 'old school'}


## Generate captions

In this section, we'll use GPT-4o mini to generate an image description and then use a few-shot examples approach with GPT-4-turbo to generate captions from the images.

If few-shot examples are not enough for your use case, consider fine-tuning a model to get the generated captions to match the style & tone you are targeting. 


```python
# Cleaning up dataset columns
selected_columns = ['title', 'primary_image', 'style', 'material', 'color', 'url']
df = df[selected_columns].copy()
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>title</th>
      <th>primary_image</th>
      <th>style</th>
      <th>material</th>
      <th>color</th>
      <th>url</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...</td>
      <td>https://m.media-amazon.com/images/I/416WaLx10j...</td>
      <td>Modern</td>
      <td>Metal</td>
      <td>White</td>
      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>
    </tr>
    <tr>
      <th>1</th>
      <td>subrtex Leather ding Room, Dining Chairs Set o...</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7...</td>
      <td>Black Rubber Wood</td>
      <td>Sponge</td>
      <td>Black</td>
      <td>https://www.amazon.com/dp/B0B66QHB23</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Plant Repotting Mat MUYETOL Waterproof Transpl...</td>
      <td>https://m.media-amazon.com/images/I/41RgefVq70...</td>
      <td>Modern</td>
      <td>Polyethylene</td>
      <td>Green</td>
      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Pickleball Doormat, Welcome Doormat Absorbent ...</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler...</td>
      <td>Modern</td>
      <td>Rubber</td>
      <td>A5589</td>
      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>
    </tr>
    <tr>
      <th>4</th>
      <td>JOIN IRON Foldable TV Trays for Eating Set of ...</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN...</td>
      <td>X Classic Style</td>
      <td>Iron</td>
      <td>Grey Set of 4</td>
      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "b67b02cc-60fe-4ba8-a2e7-5b6b833fa0b8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8aa53671-e7b1-4a68-bd91-3137292d6fd2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cc510171896f5f317f7d2dc28a109a6c4dec38f0a4b0ee780306a2532ec1ef1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "982686de-8463-4c71-8ed0-4dd8a9de69e3", "node_type": "1", "metadata": {}, "hash": "db39761c53bdb25b65353af7eebfb56b3520d5a799d85a0078c52425ef93e58b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15884, "end_char_idx": 17150, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Describing images with GPT-4o mini


```python
describe_system_prompt = '''
    You are a system generating descriptions for furniture items, decorative items, or furnishings on an e-commerce website.
    Provided with an image and a title, you will describe the main item that you see in the image, giving details but staying concise.
    You can describe unambiguously what the item is and its material, color, and style if clearly identifiable.
    If there are multiple items depicted, refer to the title to understand which item you should describe.
    '''

def describe_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,
    messages=[
        {
            "role": "system",
            "content": describe_system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
    max_tokens=300,
    )

    return response.choices[0].message.content
```

#### Testing on a few examples


key: _node_content
string_value:
{"id_": "982686de-8463-4c71-8ed0-4dd8a9de69e3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b67b02cc-60fe-4ba8-a2e7-5b6b833fa0b8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "18658ea2e53aefc937da48a535f50dd68f52f962ccfbe40955a66bed40d5870e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "07b9faae-47df-4ab8-892c-3670064c3634", "node_type": "1", "metadata": {}, "hash": "3206b7aa7ccc883300621a716574f0560b608bc87142bba4797285ab640a381e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15884, "end_char_idx": 20204, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Describing images with GPT-4o mini


```python
describe_system_prompt = '''
    You are a system generating descriptions for furniture items, decorative items, or furnishings on an e-commerce website.
    Provided with an image and a title, you will describe the main item that you see in the image, giving details but staying concise.
    You can describe unambiguously what the item is and its material, color, and style if clearly identifiable.
    If there are multiple items depicted, refer to the title to understand which item you should describe.
    '''

def describe_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,
    messages=[
        {
            "role": "system",
            "content": describe_system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
    max_tokens=300,
    )

    return response.choices[0].message.content
```

#### Testing on a few examples


```python
for index, row in examples.iterrows():
    print(f"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\n")
    img_description = describe_image(row['primary_image'], row['title'])
    print(f"{img_description}\n--------------------------\n")
```

    GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... - https://www.amazon.com/dp/B0CJHKVG6P :
    
    The item is a free-standing shoe rack designed for versatile use in living rooms, bathrooms, or hallways. It features a multi-layer metal structure with a sleek white finish. The rack includes eight double hooks at the top for hanging accessories like hats, bags, or scarves. Below, there are multiple shelves that provide ample space for organizing shoes, making it both functional and stylish for entryway storage.
    --------------------------
    
    subrtex Leather ding Room, Dining Chairs Set of 2,... - https://www.amazon.com/dp/B0B66QHB23 :
    
    The Subrtex Leather Dining Chairs come in a set of two, featuring a sleek black design. Each chair is upholstered in durable faux leather, offering a modern and stylish look. The high backrest is accentuated with subtle vertical stitching, while the sturdy wooden legs provide stability and support. These chairs are ideal for enhancing the aesthetic of any dining room.
    --------------------------
    
    Plant Repotting Mat MUYETOL Waterproof Transplanti... - https://www.amazon.com/dp/B0BXRTWLYK :
    
    The Plant Repotting Mat is a portable and foldable gardening accessory, measuring 26.8" x 26.8". It features a vibrant green color with black edges, designed to be waterproof for easy cleanup during soil changes. The mat provides a spacious area for repotting plants and comes with tools for transplanting. Ideal for indoor gardening, it helps keep your workspace tidy while you care for your succulents and other plants.
    --------------------------
    
    Pickleball Doormat, Welcome Doormat Absorbent Non-... - https://www.amazon.com/dp/B0C1MRB2M8 :
    
    The Pickleball Doormat features a natural coir material with a rectangular shape, measuring 16x24 inches. It showcases a playful design with the phrase "It's a good day to play PICKLEBALL" in bold black lettering, accompanied by graphic illustrations of pickleball paddles. The mat is designed to be absorbent and non-slip, making it suitable for entryways or bathrooms. Its light brown color adds a warm touch to any space.
    --------------------------
    
    JOIN IRON Foldable TV Trays for Eating Set of 4 wi... - https://www.amazon.com/dp/B0CG1N9QRC :
    
    The JOIN IRON Foldable TV Trays set includes four sleek, grey snack tables designed for convenience and space-saving. Each tray features a sturdy, flat surface supported by a durable metal frame, allowing for easy folding and storage. The minimalist design makes them ideal for small spaces, perfect for enjoying meals or snacks while watching TV. The set also comes with a stand for organized storage when not in use.
    --------------------------


key: _node_content
string_value:
{"id_": "07b9faae-47df-4ab8-892c-3670064c3634", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "982686de-8463-4c71-8ed0-4dd8a9de69e3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "192950bf3f9d4d2f337b5f0fe212bc56a7d6a81d4f2b1a14d0b836b2b20809da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28c7560f-49fc-4a98-8cfd-24045cf764e1", "node_type": "1", "metadata": {}, "hash": "149efdecaa0a6e636983084638984687b2a3357a614397e9110f6081cf0af020", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20212, "end_char_idx": 23971, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Turning descriptions into captions
Using a few-shot examples approach to turn a long description into a short image caption


```python
caption_system_prompt = '''
Your goal is to generate short, descriptive captions for images of furniture items, decorative items, or furnishings based on an image description.
You will be provided with a description of an item image and you will output a caption that captures the most important information about the item.
Your generated caption should be short (1 sentence), and include the most relevant information about the item.
The most important information could be: the type of the item, the style (if mentioned), the material if especially relevant and any distinctive features.
'''

few_shot_examples = [
    {
        "description": "This is a multi-layer metal shoe rack featuring a free-standing design. It has a clean, white finish that gives it a modern and versatile look, suitable for various home decors. The rack includes several horizontal shelves dedicated to organizing shoes, providing ample space for multiple pairs. Above the shoe storage area, there are 8 double hooks arranged in two rows, offering additional functionality for hanging items such as hats, scarves, or bags. The overall structure is sleek and space-saving, making it an ideal choice for placement in living rooms, bathrooms, hallways, or entryways where efficient use of space is essential.",
        "caption": "White metal free-standing shoe rack"
    },
    {
        "description": "The image shows a set of two dining chairs in black. These chairs are upholstered in a leather-like material, giving them a sleek and sophisticated appearance. The design features straight lines with a slight curve at the top of the high backrest, which adds a touch of elegance. The chairs have a simple, vertical stitching detail on the backrest, providing a subtle decorative element. The legs are also black, creating a uniform look that would complement a contemporary dining room setting. The chairs appear to be designed for comfort and style, suitable for both casual and formal dining environments.",
        "caption": "Set of 2 modern black leather dining chairs"
    },
    {
        "description": "This is a square plant repotting mat designed for indoor gardening tasks such as transplanting and changing soil for plants. It measures 26.8 inches by 26.8 inches and is made from a waterproof material, which appears to be a durable, easy-to-clean fabric in a vibrant green color. The edges of the mat are raised with integrated corner loops, likely to keep soil and water contained during gardening activities. The mat is foldable, enhancing its portability, and can be used as a protective surface for various gardening projects, including working with succulents. It's a practical accessory for garden enthusiasts and makes for a thoughtful gift for those who enjoy indoor plant care.",
        "caption": "Waterproof square plant repotting mat"
    }
]

formatted_examples = [[{
    "role": "user",
    "content": ex['description']
},
{
    "role": "assistant", 
    "content": ex['caption']
}]
    for ex in few_shot_examples
]

formatted_examples = [i for ex in formatted_examples for i in ex]
```


```python
def caption_image(description, model="gpt-4o-mini"):
    messages = formatted_examples
    messages.insert(0, 
        {
            "role": "system",
            "content": caption_system_prompt
        })
    messages.append(
        {
            "role": "user",
            "content": description
        })
    response = client.chat.completions.create(
    model=model,
    temperature=0.2,
    messages=messages
    )

    return response.choices[0].message.content
```

#### Testing on a few examples


key: _node_content
string_value:
{"id_": "28c7560f-49fc-4a98-8cfd-24045cf764e1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "07b9faae-47df-4ab8-892c-3670064c3634", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5d35d7a7c8fd2f0b24ac8862eeab02d74178aab1618a373d2181cba8130f0536", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "527eb745-965b-4906-a82e-656f6783217a", "node_type": "1", "metadata": {}, "hash": "16087b068e485932bc2ca4c49e815813d42f45abd758cc9ebe1c6aec1269e051", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23453, "end_char_idx": 27003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def caption_image(description, model="gpt-4o-mini"):
    messages = formatted_examples
    messages.insert(0, 
        {
            "role": "system",
            "content": caption_system_prompt
        })
    messages.append(
        {
            "role": "user",
            "content": description
        })
    response = client.chat.completions.create(
    model=model,
    temperature=0.2,
    messages=messages
    )

    return response.choices[0].message.content
```

#### Testing on a few examples


```python
examples = df.iloc[5:8]
```


```python
for index, row in examples.iterrows():
    print(f"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\n")
    img_description = describe_image(row['primary_image'], row['title'])
    print(f"{img_description}\n--------------------------\n")
    img_caption = caption_image(img_description)
    print(f"{img_caption}\n--------------------------\n")
```

    LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor... - https://www.amazon.com/dp/B0C9WYYFLB :
    
    The LOVMOR 30'' Bathroom Vanity Sink Base Cabinet features a classic design with a rich brown finish. It includes three drawers on the left side for ample storage, complemented by a spacious cabinet door on the right. The cabinet is constructed with detailed paneling, adding a touch of elegance, making it suitable for bathrooms, kitchens, laundry rooms, and more. Its versatile style allows it to blend seamlessly into various decor themes.
    --------------------------
    
    Classic 30'' brown bathroom vanity sink base cabinet with storage drawers.
    --------------------------
    
    Folews Bathroom Organizer Over The Toilet Storage,... - https://www.amazon.com/dp/B09NZY3R1T :
    
    The Folews Bathroom Organizer is a freestanding, 4-tier storage rack designed to fit over a toilet. It features a sleek black metal frame with adjustable shelves, allowing for customizable storage options. The shelves are made of wire, providing a modern look while ensuring durability. This organizer includes baskets for additional storage and is ideal for maximizing bathroom space by holding toiletries, towels, and other essentials. Its design is both functional and stylish, making it a great addition to any bathroom.
    --------------------------
    
    Freestanding 4-tier black metal bathroom organizer with adjustable wire shelves and baskets.
    --------------------------
    
    GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... - https://www.amazon.com/dp/B0CJHKVG6P :
    
    The GOYMFK Free Standing Shoe Rack is a versatile storage solution designed for various spaces like living rooms, bathrooms, or hallways. It features a multi-layer metal construction with a sleek white finish. The rack includes eight double hooks at the top for hanging items such as hats, bags, or scarves. Below, there are multiple shelves for organizing shoes, accommodating various styles and sizes. Its modern design combines functionality with a clean aesthetic, making it a practical addition to any home.
    --------------------------
    
    Versatile white metal free-standing shoe rack with hooks and multiple shelves.
    --------------------------
    


## Image search

In this section, we will use generated keywords and captions to search items that match a given input, either text or image.

We will leverage our embeddings model to generate embeddings for the keywords and captions and compare them to either input text or the generated caption from an input image.


key: _node_content
string_value:
{"id_": "527eb745-965b-4906-a82e-656f6783217a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28c7560f-49fc-4a98-8cfd-24045cf764e1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "58dbde861e9afbd6189f01770ae8fe14ef757234dfc8c1c68624ee4fb7582454", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6d877f1-71b1-4f2b-9e3e-b3e0873f5034", "node_type": "1", "metadata": {}, "hash": "48c6b31bc160182ee09b0a9c789fb8a8b5d868759af12f4abdab20c78eb386c8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Image search

In this section, we will use generated keywords and captions to search items that match a given input, either text or image.

We will leverage our embeddings model to generate embeddings for the keywords and captions and compare them to either input text or the generated caption from an input image.


```python
# Df we'll use to compare keywords
df_keywords = pd.DataFrame(columns=['keyword', 'embedding'])
df['keywords'] = ''
df['img_description'] = ''
df['caption'] = ''
```


```python
# Function to replace a keyword with an existing keyword if it's too similar
def get_keyword(keyword, df_keywords, threshold = 0.6):
    embedded_value = get_embedding(keyword)
    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    sorted_keywords = df_keywords.copy().sort_values('similarity', ascending=False)
    if len(sorted_keywords) > 0 :
        most_similar = sorted_keywords.iloc[0]
        if most_similar['similarity'] > threshold:
            print(f"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'")
            return most_similar['keyword']
    new_keyword = {
        'keyword': keyword,
        'embedding': embedded_value
    }
    df_keywords = pd.concat([df_keywords, pd.DataFrame([new_keyword])], ignore_index=True)
    return keyword
```

### Preparing the dataset


```python
import ast

def tag_and_caption(row):
    keywords = analyze_image(row['primary_image'], row['title'])
    try:
        keywords = ast.literal_eval(keywords)
        mapped_keywords = [get_keyword(k, df_keywords) for k in keywords]
    except Exception as e:
        print(f"Error parsing keywords: {keywords}")
        mapped_keywords = []
    img_description = describe_image(row['primary_image'], row['title'])
    caption = caption_image(img_description)
    return {
        'keywords': mapped_keywords,
        'img_description': img_description,
        'caption': caption
    }

```


```python
df.shape
```




    (312, 9)



Processing all 312 lines of the dataset will take a while.
To test out the idea, we will only run it on the first 50 lines: this takes ~20 mins. 
Feel free to skip this step and load the already processed dataset (see below).


```python
# Running on first 50 lines
for index, row in df[:50].iterrows():
    print(f"{index} - {row['title'][:50]}{'.' if len(row['title']) > 50 else ''}")
    updates = tag_and_caption(row)
    df.loc[index, updates.keys()] = updates.values()
```

    0 - GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me.
    1 - subrtex Leather ding Room, Dining Chairs Set of 2,
    2 - Plant Repotting Mat MUYETOL Waterproof Transplanti.
    3 - Pickleball Doormat, Welcome Doormat Absorbent Non-.
    4 - JOIN IRON Foldable TV Trays for Eating Set of 4 wi.
    5 - LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor.
    6 - Folews Bathroom Organizer Over The Toilet Storage,
    7 - GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me.
    8 - subrtex Leather ding Room,


key: _node_content
string_value:
{"id_": "d6d877f1-71b1-4f2b-9e3e-b3e0873f5034", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "527eb745-965b-4906-a82e-656f6783217a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f7aefc63083622ae59ea3150f2a0f550a326260725db57b4d0fbe293b1f46dbd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c072ee29-a994-4c4d-ae9f-23316977c4d3", "node_type": "1", "metadata": {}, "hash": "c0b225f0750c9b6bd820b390704e862549f24e8462b7a269543c51c2ca887ea8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
df.shape
```




    (312, 9)



Processing all 312 lines of the dataset will take a while.
To test out the idea, we will only run it on the first 50 lines: this takes ~20 mins. 
Feel free to skip this step and load the already processed dataset (see below).


```python
# Running on first 50 lines
for index, row in df[:50].iterrows():
    print(f"{index} - {row['title'][:50]}{'.' if len(row['title']) > 50 else ''}")
    updates = tag_and_caption(row)
    df.loc[index, updates.keys()] = updates.values()
```

    0 - GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me.
    1 - subrtex Leather ding Room, Dining Chairs Set of 2,
    2 - Plant Repotting Mat MUYETOL Waterproof Transplanti.
    3 - Pickleball Doormat, Welcome Doormat Absorbent Non-.
    4 - JOIN IRON Foldable TV Trays for Eating Set of 4 wi.
    5 - LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor.
    6 - Folews Bathroom Organizer Over The Toilet Storage,
    7 - GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me.
    8 - subrtex Leather ding Room, Dining Chairs Set of 2,
    9 - Plant Repotting Mat MUYETOL Waterproof Transplanti.
    10 - Pickleball Doormat, Welcome Doormat Absorbent Non-.
    11 - JOIN IRON Foldable TV Trays for Eating Set of 4 wi.
    12 - LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor.
    13 - Folews Bathroom Organizer Over The Toilet Storage,
    14 - Lerliuo Nightstand, Side Table, Industrial Bedside.
    15 - Boss Office Products Any Task Mid-Back Task Chair .
    16 - Kingston Brass BA1752BB Heritage 18-Inch Towel-Bar.
    17 - Chief Mfg.Swing-Arm Wall Mount Hardware Mount Blac.
    18 - DOMYDEVM Black End Table, Nightstand with Charging.
    19 - LASCO 35-5019 Hallmack Style 24-Inch Towel Bar Acc.
    20 - Table-Mate II PRO TV Tray Table - Folding Table wi.
    21 - EGFheal White Dress Up Storage
    22 - Caroline's Treasures PPD3013JMAT Enchanted Garden .
    23 - Leick Home 70007-WTGD Mixed Metal and Wood Stepped.
    24 - Caroline's Treasures CK3435MAT Bichon Frise Doorma.
    25 - Wildkin Kids Canvas Sling Bookshelf with Storage f.
    26 - Gbuzozie 38L Round Laundry Hamper Cute Mermaid Gir.
    27 - Tiita Comfy Saucer Chair, Soft Faux Fur Oversized .
    28 - Summer Desk Decor,Welcome Summer Wood Block Sign D.
    29 - Homebeez 39.1" Length Bedroom Storage Bench, End B.
    30 - Flash Furniture Webb Commercial Grade 24" Round Bl.
    31 - Mellow 2 Inch Ventilated Memory Foam Mattress Topp.


key: _node_content
string_value:
{"id_": "c072ee29-a994-4c4d-ae9f-23316977c4d3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6d877f1-71b1-4f2b-9e3e-b3e0873f5034", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "17b1e1547bf60ba95e13f1086fe3f8dae4b26c5c46b1fef6a9a9e17c065b3305", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0dd6ad69-63d4-46b5-86a6-e93276132c4a", "node_type": "1", "metadata": {}, "hash": "5b6d284b05c0d2e50df9709603e6226ca22438cd7c92030934a481f44063e7d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
11 - JOIN IRON Foldable TV Trays for Eating Set of 4 wi.
    12 - LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor.
    13 - Folews Bathroom Organizer Over The Toilet Storage,
    14 - Lerliuo Nightstand, Side Table, Industrial Bedside.
    15 - Boss Office Products Any Task Mid-Back Task Chair .
    16 - Kingston Brass BA1752BB Heritage 18-Inch Towel-Bar.
    17 - Chief Mfg.Swing-Arm Wall Mount Hardware Mount Blac.
    18 - DOMYDEVM Black End Table, Nightstand with Charging.
    19 - LASCO 35-5019 Hallmack Style 24-Inch Towel Bar Acc.
    20 - Table-Mate II PRO TV Tray Table - Folding Table wi.
    21 - EGFheal White Dress Up Storage
    22 - Caroline's Treasures PPD3013JMAT Enchanted Garden .
    23 - Leick Home 70007-WTGD Mixed Metal and Wood Stepped.
    24 - Caroline's Treasures CK3435MAT Bichon Frise Doorma.
    25 - Wildkin Kids Canvas Sling Bookshelf with Storage f.
    26 - Gbuzozie 38L Round Laundry Hamper Cute Mermaid Gir.
    27 - Tiita Comfy Saucer Chair, Soft Faux Fur Oversized .
    28 - Summer Desk Decor,Welcome Summer Wood Block Sign D.
    29 - Homebeez 39.1" Length Bedroom Storage Bench, End B.
    30 - Flash Furniture Webb Commercial Grade 24" Round Bl.
    31 - Mellow 2 Inch Ventilated Memory Foam Mattress Topp.
    32 - CangLong Mid Century Modern Side Chair with Wood L.
    33 - HomePop Metal Accent Table Triangle Base Round Mir.
    34 - MAEPA RV Shoe Storage for Bedside - 8 Extra Large .
    35 - NearMoon Hand Towel Holder/Towel Ring - Bathroom T.
    36 - FLYJOE Narrow Side Table with PU Leather Magazine .
    37 - HomePop Home Decor | K2380-YDQY-2 | Luxury Large F.
    38 - Moroccan Leather Pouf Ottoman for Living Room - Ro.
    39 - AnyDesign Christmas Welcome Doormat Decorative Xma.
    40 - GXFC ZHAO Welcome Funny Door Mat Shoes and Bras Of.
    41 - LEASYLIFE Black Metal Trash can,10L/2.6GAL,Open To.
    42 - Solid Wood Wine Cabinet, bar Rack - Home Wood Furn.
    43 - Black Leather Office Chair Mid Back Leather Desk C.
    44 - Convenience Concepts Tucson Flip Top End Table wit.
    45 - 3-Tier Kitchen Storage Cart with Handle, Multifunc.
    46 - Mimoglad Office Chair, High Back Ergonomic Desk Ch.
    47 - Let the Adventure Begin Door Mat 17"x30" Decorativ.
    48 - 1 Pack Adjustable Height Center Support Leg for Be.
    49 - Stylo Culture Traditional Cotton Patchwork Embroid.



```python
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;


key: _node_content
string_value:
{"id_": "0dd6ad69-63d4-46b5-86a6-e93276132c4a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c072ee29-a994-4c4d-ae9f-23316977c4d3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "11dffdd6384c80795efb91b4d169796f73b4e59f8c15fe6488627850bdfda630", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fe73aca-275c-4d7b-85ee-785b6e7a4c0d", "node_type": "1", "metadata": {}, "hash": "b59159eeac6992e2950a632133b92117feeba7d135588254eb02637322d074d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
31 - Mellow 2 Inch Ventilated Memory Foam Mattress Topp.
    32 - CangLong Mid Century Modern Side Chair with Wood L.
    33 - HomePop Metal Accent Table Triangle Base Round Mir.
    34 - MAEPA RV Shoe Storage for Bedside - 8 Extra Large .
    35 - NearMoon Hand Towel Holder/Towel Ring - Bathroom T.
    36 - FLYJOE Narrow Side Table with PU Leather Magazine .
    37 - HomePop Home Decor | K2380-YDQY-2 | Luxury Large F.
    38 - Moroccan Leather Pouf Ottoman for Living Room - Ro.
    39 - AnyDesign Christmas Welcome Doormat Decorative Xma.
    40 - GXFC ZHAO Welcome Funny Door Mat Shoes and Bras Of.
    41 - LEASYLIFE Black Metal Trash can,10L/2.6GAL,Open To.
    42 - Solid Wood Wine Cabinet, bar Rack - Home Wood Furn.
    43 - Black Leather Office Chair Mid Back Leather Desk C.
    44 - Convenience Concepts Tucson Flip Top End Table wit.
    45 - 3-Tier Kitchen Storage Cart with Handle, Multifunc.
    46 - Mimoglad Office Chair, High Back Ergonomic Desk Ch.
    47 - Let the Adventure Begin Door Mat 17"x30" Decorativ.
    48 - 1 Pack Adjustable Height Center Support Leg for Be.
    49 - Stylo Culture Traditional Cotton Patchwork Embroid.



```python
df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>title</th>
      <th>primary_image</th>
      <th>style</th>
      <th>material</th>
      <th>color</th>
      <th>url</th>
      <th>keywords</th>
      <th>img_description</th>
      <th>caption</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye.</td>
      <td>https://m.media-amazon.com/images/I/416WaLx10j.</td>
      <td>Modern</td>
      <td>Metal</td>
      <td>White</td>
      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>
      <td>[shoe rack, metal, white, multi-layer, hooks]</td>
      <td>The GOYMFK Free Standing Shoe Rack is a versat.</td>
      <td>Sleek white multi-layer metal free-standing sh.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>subrtex Leather ding Room, Dining Chairs Set o.</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7.


key: _node_content
string_value:
{"id_": "8fe73aca-275c-4d7b-85ee-785b6e7a4c0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0dd6ad69-63d4-46b5-86a6-e93276132c4a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1a4ec6997941bb6fe2ab4fb1d1ea715513c456306480d3f17ffc27b7c86cc79e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e1cb56ac-59bc-4da7-bfe5-7dfdb2639bd0", "node_type": "1", "metadata": {}, "hash": "7d09a4b07eeb1646426980e88f1d01fa49c04d9b8bb39c23b5c9dbe0d369034e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>title</th>
      <th>primary_image</th>
      <th>style</th>
      <th>material</th>
      <th>color</th>
      <th>url</th>
      <th>keywords</th>
      <th>img_description</th>
      <th>caption</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye.</td>
      <td>https://m.media-amazon.com/images/I/416WaLx10j.</td>
      <td>Modern</td>
      <td>Metal</td>
      <td>White</td>
      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>
      <td>[shoe rack, metal, white, multi-layer, hooks]</td>
      <td>The GOYMFK Free Standing Shoe Rack is a versat.</td>
      <td>Sleek white multi-layer metal free-standing sh.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>subrtex Leather ding Room, Dining Chairs Set o.</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7.</td>
      <td>Black Rubber Wood</td>
      <td>Sponge</td>
      <td>Black</td>
      <td>https://www.amazon.com/dp/B0B66QHB23</td>
      <td>[dining chair, leather, black]</td>
      <td>The Subrtex Leather Dining Chairs come in a se.</td>
      <td>Set of 2 modern black faux leather dining chai.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Plant Repotting Mat MUYETOL Waterproof Transpl.</td>
      <td>https://m.media-amazon.com/images/I/41RgefVq70.</td>
      <td>Modern</td>
      <td>Polyethylene</td>
      <td>Green</td>
      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>
      <td>[repotting mat, waterproof, portable, foldable.</td>
      <td>The Plant Repotting Mat is a portable and fold.</td>
      <td>Vibrant green waterproof plant repotting mat</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Pickleball Doormat, Welcome Doormat Absorbent .</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler.</td>
      <td>Modern</td>
      <td>Rubber</td>
      <td>A5589</td>
      <td>https://www.amazon.


key: _node_content
string_value:
{"id_": "e1cb56ac-59bc-4da7-bfe5-7dfdb2639bd0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fe73aca-275c-4d7b-85ee-785b6e7a4c0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c09383b3be0ce737db6c22f82917598b8e2918ee3f922194902c4d926b45a2b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d99e04ef-758c-4a12-8916-16b8cbefa055", "node_type": "1", "metadata": {}, "hash": "014a4dd705886fba849f80e7d59559488811cc54d3b5ea22c616706a25dd83b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Dining Chairs Set o.</td>
      <td>https://m.media-amazon.com/images/I/31SejUEWY7.</td>
      <td>Black Rubber Wood</td>
      <td>Sponge</td>
      <td>Black</td>
      <td>https://www.amazon.com/dp/B0B66QHB23</td>
      <td>[dining chair, leather, black]</td>
      <td>The Subrtex Leather Dining Chairs come in a se.</td>
      <td>Set of 2 modern black faux leather dining chai.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Plant Repotting Mat MUYETOL Waterproof Transpl.</td>
      <td>https://m.media-amazon.com/images/I/41RgefVq70.</td>
      <td>Modern</td>
      <td>Polyethylene</td>
      <td>Green</td>
      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>
      <td>[repotting mat, waterproof, portable, foldable.</td>
      <td>The Plant Repotting Mat is a portable and fold.</td>
      <td>Vibrant green waterproof plant repotting mat</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Pickleball Doormat, Welcome Doormat Absorbent .</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler.</td>
      <td>Modern</td>
      <td>Rubber</td>
      <td>A5589</td>
      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>
      <td>[doormat, absorbent, non-slip, coconut fiber, .</td>
      <td>The Pickleball Doormat is a charming welcome m.</td>
      <td>Coir welcome mat featuring a playful "It's a g.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>JOIN IRON Foldable TV Trays for Eating Set of .</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN.</td>
      <td>X Classic Style</td>
      <td>Iron</td>
      <td>Grey Set of 4</td>
      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>
      <td>[tv tray, foldable, metal, grey]</td>
      <td>The JOIN IRON Foldable TV Tray Set includes fo.</td>
      <td>Set of 4 foldable grey TV trays with durable b.</td>
    </tr>
  </tbody>
</table>
</div>




```python
data_path = "data/items_tagged_and_captioned.csv"
```


```python
# Saving locally for later - optional: do not execute if you prefer to use the provided file
df.to_csv(data_path, index=False)
```


key: _node_content
string_value:
{"id_": "d99e04ef-758c-4a12-8916-16b8cbefa055", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e1cb56ac-59bc-4da7-bfe5-7dfdb2639bd0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f2f7b41cce6db054cef1a5653624860d6e5b598a792fed0b2aa40370eace8098", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f1d172b-d63f-45db-b6e4-4dc40eb58699", "node_type": "1", "metadata": {}, "hash": "2fb8c9d17474f1e6191d5282584addebf1a65f30053a4879674fb7187b156035", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Welcome Doormat Absorbent .</td>
      <td>https://m.media-amazon.com/images/I/61vz1Igler.</td>
      <td>Modern</td>
      <td>Rubber</td>
      <td>A5589</td>
      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>
      <td>[doormat, absorbent, non-slip, coconut fiber, .</td>
      <td>The Pickleball Doormat is a charming welcome m.</td>
      <td>Coir welcome mat featuring a playful "It's a g.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>JOIN IRON Foldable TV Trays for Eating Set of .</td>
      <td>https://m.media-amazon.com/images/I/41p4d4VJnN.</td>
      <td>X Classic Style</td>
      <td>Iron</td>
      <td>Grey Set of 4</td>
      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>
      <td>[tv tray, foldable, metal, grey]</td>
      <td>The JOIN IRON Foldable TV Tray Set includes fo.</td>
      <td>Set of 4 foldable grey TV trays with durable b.</td>
    </tr>
  </tbody>
</table>
</div>




```python
data_path = "data/items_tagged_and_captioned.csv"
```


```python
# Saving locally for later - optional: do not execute if you prefer to use the provided file
df.to_csv(data_path, index=False)
```


```python
# Optional: load data from saved file if you haven't processed the whole dataset
df = pd.read_csv(data_path)
```

### Embedding captions and keywords
We can now use the generated captions and keywords to match relevant content to an input text query or caption. 
To do this, we will embed a combination of keywords + captions.
Note: creating the embeddings will take ~3 mins to run. Feel free to load the pre-processed dataset (see below).


```python
df_search = df.copy()
```


```python
def embed_tags_caption(x):
    if x['caption'] != '':
        try:
            keywords_string = ",".join(k for k in x['keywords']) + '\n'
            content = keywords_string + x['caption']
            embedding = get_embedding(content)
            return embedding
        except Exception as e:
            print(f"Error creating embedding for {x}: {e}")
```


```python
df_search['embedding'] = df_search.apply(lambda x: embed_tags_caption(x), axis=1)
```

    Error creating embedding for title              Suptsifira Shoe storage box, 24 Packs Shoe Box.
    primary_image      https://m.media-amazon.com/images/I/51enKGSxK8.
    style                                                            NaN
    material                                                   Porcelain
    color                                                          White
    url                             https://www.amazon.com/dp/B0BZ85JVBN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 50, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Wellynap Computer Desk,31.5 inches Folding Tab.
    primary_image      https://m.media-amazon.com/images/I/51pO-N48te.


key: _node_content
string_value:
{"id_": "5f1d172b-d63f-45db-b6e4-4dc40eb58699", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d99e04ef-758c-4a12-8916-16b8cbefa055", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "787144395df29c803dfe55f1ed70faae58a93ca7260961557d6fe2e6f0517cb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f7af168-22f8-49a6-acd0-5ff703f55045", "node_type": "1", "metadata": {}, "hash": "99972be17c2f6140863e1ce8c554239bafd4bf59b6e1a6db2c69027661cf0c75", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Optional: load data from saved file if you haven't processed the whole dataset
df = pd.read_csv(data_path)
```

### Embedding captions and keywords
We can now use the generated captions and keywords to match relevant content to an input text query or caption. 
To do this, we will embed a combination of keywords + captions.
Note: creating the embeddings will take ~3 mins to run. Feel free to load the pre-processed dataset (see below).


```python
df_search = df.copy()
```


```python
def embed_tags_caption(x):
    if x['caption'] != '':
        try:
            keywords_string = ",".join(k for k in x['keywords']) + '\n'
            content = keywords_string + x['caption']
            embedding = get_embedding(content)
            return embedding
        except Exception as e:
            print(f"Error creating embedding for {x}: {e}")
```


```python
df_search['embedding'] = df_search.apply(lambda x: embed_tags_caption(x), axis=1)
```

    Error creating embedding for title              Suptsifira Shoe storage box, 24 Packs Shoe Box.
    primary_image      https://m.media-amazon.com/images/I/51enKGSxK8.
    style                                                            NaN
    material                                                   Porcelain
    color                                                          White
    url                             https://www.amazon.com/dp/B0BZ85JVBN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 50, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Wellynap Computer Desk,31.5 inches Folding Tab.
    primary_image      https://m.media-amazon.com/images/I/51pO-N48te.
    style                                                         Modern
    material                                                        Wood
    color                                                   Teak & Black
    url                             https://www.amazon.com/dp/B0CFL2G31X
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 51, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Smlttel Gold Clothing Rack With Shelves, Gold .
    primary_image      https://m.media-amazon.com/images/I/41aRwocdfA.
    style                                                         Modern
    material                                                       Metal
    color                                                         C gold
    url                             https://www.amazon.com/dp/B0B93TC1Z8
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 52, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Franklin Sports NFL Storage Ottoman + Containe.
    primary_image      https://m.media-amazon.com/images/I/31ptZB+wS-.
    style              Team Licensed Storage Ottoman with Detachable Lid
    material                                                      Fabric
    color                                                     Team Color
    url                             https://www.amazon.com/dp/B0787KRJ8S
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 53, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Honey-Can-Do 3-Tier Nesting Bamboo Shoe Rack S.
    primary_image      https://m.media-amazon.com/images/I/51GnnjKaVs.
    style                                                           Shoe
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "4f7af168-22f8-49a6-acd0-5ff703f55045", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f1d172b-d63f-45db-b6e4-4dc40eb58699", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "678bf1ed128dd7efeca77ff4f92c6a3c13f696d06c77dd2a539679cacd07fa11", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "feb07503-1de7-4e8a-a753-077102f836d0", "node_type": "1", "metadata": {}, "hash": "7b6494f1f6869d1840617de35695889bdb5412a916d9902a584ce1f158758fab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/51pO-N48te.
    style                                                         Modern
    material                                                        Wood
    color                                                   Teak & Black
    url                             https://www.amazon.com/dp/B0CFL2G31X
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 51, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Smlttel Gold Clothing Rack With Shelves, Gold .
    primary_image      https://m.media-amazon.com/images/I/41aRwocdfA.
    style                                                         Modern
    material                                                       Metal
    color                                                         C gold
    url                             https://www.amazon.com/dp/B0B93TC1Z8
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 52, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Franklin Sports NFL Storage Ottoman + Containe.
    primary_image      https://m.media-amazon.com/images/I/31ptZB+wS-.
    style              Team Licensed Storage Ottoman with Detachable Lid
    material                                                      Fabric
    color                                                     Team Color
    url                             https://www.amazon.com/dp/B0787KRJ8S
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 53, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Honey-Can-Do 3-Tier Nesting Bamboo Shoe Rack S.
    primary_image      https://m.media-amazon.com/images/I/51GnnjKaVs.
    style                                                           Shoe
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B08WRLKR7T
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 54, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Furnistar 15.9 inch Modern Round Velvet Storag.
    primary_image      https://m.media-amazon.com/images/I/31IBS5mzYS.
    style                                                         Modern
    material                                                        Wood
    color                                                           Grey
    url                             https://www.amazon.com/dp/B0C4NT8N8C
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 55, dtype: object: 'float' object is not iterable
    Error creating embedding for title              AMHANCIBLE C Shaped Side Table, End Tables Set.
    primary_image      https://m.media-amazon.com/images/I/41qDAGoNCr.
    style                                                   Straight Leg
    material                                             Engineered Wood
    color                                                          Black
    url                             https://www.amazon.com/dp/B0BT9SVN1V
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 56, dtype: object: 'float' object is not iterable
    Error creating embedding for title              LONGWIN Black Hanging Wall Round Mirror Decor .
    primary_image      https://m.media-amazon.com/images/I/41kC6cU5HX.
    style                                                         Modern
    material                                                Glass, Metal
    color                                                          Black
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "feb07503-1de7-4e8a-a753-077102f836d0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f7af168-22f8-49a6-acd0-5ff703f55045", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "008366e71efbeecde81e1ee751251c8ac6d2a1cf6680d2373457788e2f8c0537", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "413071da-fd17-49c9-837d-5712dc3c6785", "node_type": "1", "metadata": {}, "hash": "edb4b16439e9d3bcabd4b4bc78cee1a76f107648db077a2775fe2e8bbb1a71df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/51GnnjKaVs.
    style                                                           Shoe
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B08WRLKR7T
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 54, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Furnistar 15.9 inch Modern Round Velvet Storag.
    primary_image      https://m.media-amazon.com/images/I/31IBS5mzYS.
    style                                                         Modern
    material                                                        Wood
    color                                                           Grey
    url                             https://www.amazon.com/dp/B0C4NT8N8C
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 55, dtype: object: 'float' object is not iterable
    Error creating embedding for title              AMHANCIBLE C Shaped Side Table, End Tables Set.
    primary_image      https://m.media-amazon.com/images/I/41qDAGoNCr.
    style                                                   Straight Leg
    material                                             Engineered Wood
    color                                                          Black
    url                             https://www.amazon.com/dp/B0BT9SVN1V
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 56, dtype: object: 'float' object is not iterable
    Error creating embedding for title              LONGWIN Black Hanging Wall Round Mirror Decor .
    primary_image      https://m.media-amazon.com/images/I/41kC6cU5HX.
    style                                                         Modern
    material                                                Glass, Metal
    color                                                          Black
    url                             https://www.amazon.com/dp/B094F897P3
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 57, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Need Fold Wall Mounted Workbench Folding Wall .
    primary_image      https://m.media-amazon.com/images/I/31SqvdFCut.
    style                                                         Modern
    material                                                       Metal
    color               Teak Color Desktop & Warm White Folding Brackets
    url                             https://www.amazon.com/dp/B00UV7B29A
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 58, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Big Joe Fuf XL Cover Only Machine Washable, Gr.
    primary_image      https://m.media-amazon.com/images/I/21ysztDdCY.
    style                                                          Plush
    material                                                         NaN
    color                                                           Grey
    url                             https://www.amazon.com/dp/B08T7JP8ZN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 59, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Plymor Rectangle 5mm Beveled Glass Mirror, 6 i.
    primary_image      https://m.media-amazon.com/images/I/31wigA5chu.
    style                                                            NaN
    material                                                       Glass
    color                                                         Silver
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "413071da-fd17-49c9-837d-5712dc3c6785", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "feb07503-1de7-4e8a-a753-077102f836d0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8b6e919d618a55fba46ab0306e02b4bef489442f0dac012939772fdfc93a49e5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6ae6bba1-fc34-480d-ad31-a7ce08414ed7", "node_type": "1", "metadata": {}, "hash": "ac6e3cf0084aeaa1fa0958445849c1f7b88dafc05316c238f6c82beb5b2dd3a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
primary_image      https://m.media-amazon.com/images/I/41kC6cU5HX.
    style                                                         Modern
    material                                                Glass, Metal
    color                                                          Black
    url                             https://www.amazon.com/dp/B094F897P3
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 57, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Need Fold Wall Mounted Workbench Folding Wall .
    primary_image      https://m.media-amazon.com/images/I/31SqvdFCut.
    style                                                         Modern
    material                                                       Metal
    color               Teak Color Desktop & Warm White Folding Brackets
    url                             https://www.amazon.com/dp/B00UV7B29A
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 58, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Big Joe Fuf XL Cover Only Machine Washable, Gr.
    primary_image      https://m.media-amazon.com/images/I/21ysztDdCY.
    style                                                          Plush
    material                                                         NaN
    color                                                           Grey
    url                             https://www.amazon.com/dp/B08T7JP8ZN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 59, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Plymor Rectangle 5mm Beveled Glass Mirror, 6 i.
    primary_image      https://m.media-amazon.com/images/I/31wigA5chu.
    style                                                            NaN
    material                                                       Glass
    color                                                         Silver
    url                             https://www.amazon.com/dp/B09F3SGZ8Y
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 60, dtype: object: 'float' object is not iterable
    Error creating embedding for title              TIMCORR CD Case DVD Holder Storage: 144 Capaci.
    primary_image      https://m.media-amazon.com/images/I/411Q2ETwel.
    style                                                       Portable
    material                           EVA + PVC + PP + Non-woven fabric
    color                                                          Black
    url                             https://www.amazon.com/dp/B0B19ZGGXC
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 61, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Ginger Cayden Closed Towel Ring - 4905/SN - Sa.
    primary_image      https://m.media-amazon.com/images/I/31LNv7QILd.
    style                                                            NaN
    material                                                       Brass
    color                                                   Satin Nickel
    url                             https://www.amazon.com/dp/B00U0ECLG2
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 62, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Brightify Black Bathroom Mirrors for Wall, 24 .
    primary_image      https://m.media-amazon.com/images/I/510A0nIdGZ.
    style                                                         Modern
    material                                                    Aluminum
    color                                                          Black
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "6ae6bba1-fc34-480d-ad31-a7ce08414ed7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "413071da-fd17-49c9-837d-5712dc3c6785", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "75309ad7caa8f8d9d91ba56512408f16751f01bf3389a3179c54a3bb8e2935ce", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30ee7b9d-3cf7-4d58-a323-67d841a17527", "node_type": "1", "metadata": {}, "hash": "dd0d0be6dfe672969a56f41f0fe9424944e9b35badc9924269ab0e94b7b6a3bd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/31wigA5chu.
    style                                                            NaN
    material                                                       Glass
    color                                                         Silver
    url                             https://www.amazon.com/dp/B09F3SGZ8Y
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 60, dtype: object: 'float' object is not iterable
    Error creating embedding for title              TIMCORR CD Case DVD Holder Storage: 144 Capaci.
    primary_image      https://m.media-amazon.com/images/I/411Q2ETwel.
    style                                                       Portable
    material                           EVA + PVC + PP + Non-woven fabric
    color                                                          Black
    url                             https://www.amazon.com/dp/B0B19ZGGXC
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 61, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Ginger Cayden Closed Towel Ring - 4905/SN - Sa.
    primary_image      https://m.media-amazon.com/images/I/31LNv7QILd.
    style                                                            NaN
    material                                                       Brass
    color                                                   Satin Nickel
    url                             https://www.amazon.com/dp/B00U0ECLG2
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 62, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Brightify Black Bathroom Mirrors for Wall, 24 .
    primary_image      https://m.media-amazon.com/images/I/510A0nIdGZ.
    style                                                         Modern
    material                                                    Aluminum
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C2HNGCRX
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 63, dtype: object: 'float' object is not iterable
    Error creating embedding for title              SogesHome Wood Corner Cabinet Wall Corner Stor.
    primary_image      https://m.media-amazon.com/images/I/41BTUFVwm+.
    style                                                     Open Frame
    material                                                         NaN
    color                                                     White&teak
    url                             https://www.amazon.com/dp/B0C3B4D4RH
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 64, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Toy Storage for Lego Play Mat Bag - Duplo Toy .
    primary_image      https://m.media-amazon.com/images/I/51KKvmDCqB.
    style                                                            NaN
    material                                                       Nylon
    color                                                         Orange
    url                             https://www.amazon.com/dp/B0B4CL1M1M
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 65, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Flash Furniture Jefferson 2 Pk. Contemporary B.
    primary_image      https://m.media-amazon.com/images/I/41GYYVLfGj.
    style                                                   Contemporary
    material                                                         NaN
    color                                                          Brown
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "30ee7b9d-3cf7-4d58-a323-67d841a17527", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ae6bba1-fc34-480d-ad31-a7ce08414ed7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "10a905c31c8d3414d8e052003faf749ecab6871634811bc420c975de6b4de72a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3619f7a-f31c-4173-8e60-7a77effbb9ea", "node_type": "1", "metadata": {}, "hash": "3d04f02e242b2724298656e70648413be5e7a7968b48e59b9670d1b648089dff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
primary_image      https://m.media-amazon.com/images/I/510A0nIdGZ.
    style                                                         Modern
    material                                                    Aluminum
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C2HNGCRX
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 63, dtype: object: 'float' object is not iterable
    Error creating embedding for title              SogesHome Wood Corner Cabinet Wall Corner Stor.
    primary_image      https://m.media-amazon.com/images/I/41BTUFVwm+.
    style                                                     Open Frame
    material                                                         NaN
    color                                                     White&teak
    url                             https://www.amazon.com/dp/B0C3B4D4RH
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 64, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Toy Storage for Lego Play Mat Bag - Duplo Toy .
    primary_image      https://m.media-amazon.com/images/I/51KKvmDCqB.
    style                                                            NaN
    material                                                       Nylon
    color                                                         Orange
    url                             https://www.amazon.com/dp/B0B4CL1M1M
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 65, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Flash Furniture Jefferson 2 Pk. Contemporary B.
    primary_image      https://m.media-amazon.com/images/I/41GYYVLfGj.
    style                                                   Contemporary
    material                                                         NaN
    color                                                          Brown
    url                             https://www.amazon.com/dp/B00FEAN1SY
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 66, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Hong Art- Metal Mirror-Matt Black,Glass Panel .
    primary_image      https://m.media-amazon.com/images/I/31XytAHobH.
    style                                                        Classic
    material                                                       Metal
    color                                                          Black
    url                             https://www.amazon.com/dp/B08GSH4KVM
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 67, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Convenience Concepts American Heritage Round E.
    primary_image      https://m.media-amazon.com/images/I/311rmB9BDW.
    style                                                Round End Table
    material           Solid + Manufactured Wood,Particle Board/Chipb.
    color                                                           Pink
    url                             https://www.amazon.com/dp/B01B65BYYI
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 68, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Flash Furniture Diamond Black Vinyl Luxurious .
    primary_image      https://m.media-amazon.com/images/I/41LYsAMww6.
    style                                                          Fixed
    material                                                        Foam
    color                                                    Black Vinyl
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "d3619f7a-f31c-4173-8e60-7a77effbb9ea", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30ee7b9d-3cf7-4d58-a323-67d841a17527", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "62cdea5cac563f8c8c115d8d33a19e4e90c0c7e9d6ea765a44ac5b33465d0225", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e9d278b-4ee4-4de6-b7a0-1b279eb3eb6e", "node_type": "1", "metadata": {}, "hash": "b0557dba5ff9dd05dba4b874925223fc1cc0e089974190dc5b91d15f45eeeb1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Contemporary B.
    primary_image      https://m.media-amazon.com/images/I/41GYYVLfGj.
    style                                                   Contemporary
    material                                                         NaN
    color                                                          Brown
    url                             https://www.amazon.com/dp/B00FEAN1SY
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 66, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Hong Art- Metal Mirror-Matt Black,Glass Panel .
    primary_image      https://m.media-amazon.com/images/I/31XytAHobH.
    style                                                        Classic
    material                                                       Metal
    color                                                          Black
    url                             https://www.amazon.com/dp/B08GSH4KVM
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 67, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Convenience Concepts American Heritage Round E.
    primary_image      https://m.media-amazon.com/images/I/311rmB9BDW.
    style                                                Round End Table
    material           Solid + Manufactured Wood,Particle Board/Chipb.
    color                                                           Pink
    url                             https://www.amazon.com/dp/B01B65BYYI
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 68, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Flash Furniture Diamond Black Vinyl Luxurious .
    primary_image      https://m.media-amazon.com/images/I/41LYsAMww6.
    style                                                          Fixed
    material                                                        Foam
    color                                                    Black Vinyl
    url                             https://www.amazon.com/dp/B000TMHWGO
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 69, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Gatco 1918, Modern Rectangle Waste Basket, Mat.
    primary_image      https://m.media-amazon.com/images/I/31dnAVaEmv.
    style                                                      Rectangle
    material                                             Stainless Steel
    color                                                    Matte Black
    url                             https://www.amazon.com/dp/B07TXMJ5FQ
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 70, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Winrise Office Chair Ergonomic Desk Chair, Hig.
    primary_image      https://m.media-amazon.com/images/I/41hCFaVIC+.
    style                                                       Straight
    material                                                      Sponge
    color                                                        S-black
    url                             https://www.amazon.com/dp/B0CGQZBCZP
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 71, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Adeco Euro Style Fabric Arm Bench Chair Footst.
    primary_image      https://m.media-amazon.com/images/I/41hUc8c+DC.
    style                                                         Modern
    material                                             Engineered Wood
    color                                                          Brown
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "3e9d278b-4ee4-4de6-b7a0-1b279eb3eb6e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3619f7a-f31c-4173-8e60-7a77effbb9ea", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0cac5aca304a27dec54aa92d9909ca6993178ae4ec7526bb5f42472d3650731b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8af43f45-9d6c-4eb8-9b66-aebf178b9837", "node_type": "1", "metadata": {}, "hash": "04da954db1bca43bb0e36391d4b9bc4e8cfc3c08290b36f9e11dc15045501340", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
primary_image      https://m.media-amazon.com/images/I/41LYsAMww6.
    style                                                          Fixed
    material                                                        Foam
    color                                                    Black Vinyl
    url                             https://www.amazon.com/dp/B000TMHWGO
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 69, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Gatco 1918, Modern Rectangle Waste Basket, Mat.
    primary_image      https://m.media-amazon.com/images/I/31dnAVaEmv.
    style                                                      Rectangle
    material                                             Stainless Steel
    color                                                    Matte Black
    url                             https://www.amazon.com/dp/B07TXMJ5FQ
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 70, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Winrise Office Chair Ergonomic Desk Chair, Hig.
    primary_image      https://m.media-amazon.com/images/I/41hCFaVIC+.
    style                                                       Straight
    material                                                      Sponge
    color                                                        S-black
    url                             https://www.amazon.com/dp/B0CGQZBCZP
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 71, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Adeco Euro Style Fabric Arm Bench Chair Footst.
    primary_image      https://m.media-amazon.com/images/I/41hUc8c+DC.
    style                                                         Modern
    material                                             Engineered Wood
    color                                                          Brown
    url                             https://www.amazon.com/dp/B017TNJR72
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 72, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Motiv 0202/PC Sine 18-In Towel Bar, Polished C.
    primary_image      https://m.media-amazon.com/images/I/31a6GfenW0.
    style                                                            NaN
    material                                                       Brass
    color                                                  18" Towel Bar
    url                             https://www.amazon.com/dp/B001AS8D82
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 73, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Imports Décor PVC Backed Coir Doormat, Eighth .
    primary_image      https://m.media-amazon.com/images/I/51H9lDOICr.
    style                                                       Art Deco
    material                                                       Vinyl
    color                                                Black and Beige
    url                             https://www.amazon.com/dp/B08WF83LMF
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 74, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Croydex Chester Square Flexi-Fix Wall Mounted .
    primary_image      https://m.media-amazon.com/images/I/41sDO1HW2c.
    style                                                            NaN
    material                                                         NaN
    color                                                         Silver
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "8af43f45-9d6c-4eb8-9b66-aebf178b9837", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e9d278b-4ee4-4de6-b7a0-1b279eb3eb6e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "20e6be99358c25427bcc2136f699162a66021ed61f40224e1dcc570fc95e7e54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25b2cd7d-983b-431c-8ece-346522a63b2b", "node_type": "1", "metadata": {}, "hash": "eca1845bc2d5dbac4e9f9e9735c0014023c3cadea68c1cdd65721caeee0b7875", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
com/images/I/41hUc8c+DC.
    style                                                         Modern
    material                                             Engineered Wood
    color                                                          Brown
    url                             https://www.amazon.com/dp/B017TNJR72
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 72, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Motiv 0202/PC Sine 18-In Towel Bar, Polished C.
    primary_image      https://m.media-amazon.com/images/I/31a6GfenW0.
    style                                                            NaN
    material                                                       Brass
    color                                                  18" Towel Bar
    url                             https://www.amazon.com/dp/B001AS8D82
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 73, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Imports Décor PVC Backed Coir Doormat, Eighth .
    primary_image      https://m.media-amazon.com/images/I/51H9lDOICr.
    style                                                       Art Deco
    material                                                       Vinyl
    color                                                Black and Beige
    url                             https://www.amazon.com/dp/B08WF83LMF
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 74, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Croydex Chester Square Flexi-Fix Wall Mounted .
    primary_image      https://m.media-amazon.com/images/I/41sDO1HW2c.
    style                                                            NaN
    material                                                         NaN
    color                                                         Silver
    url                             https://www.amazon.com/dp/B09DGFRM4B
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 75, dtype: object: 'float' object is not iterable
    Error creating embedding for title              itbe Easy Fit Ready-to-Assemble Multipurpose O.
    primary_image      https://m.media-amazon.com/images/I/21NWASZgUV.
    style                                                     Flat Panel
    material                                                 Alloy Steel
    color                                                           Blue
    url                             https://www.amazon.com/dp/B09FR4XSCT
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 76, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Delta ARV18-DN Arvo 18-in Wall Mount Towel Bar.
    primary_image      https://m.media-amazon.com/images/I/11zzs81fXB.
    style                                 18" Towel Bar with 6" Extender
    material                                     Multiple Base Materials
    color                                      Spotshield Brushed Nickel
    url                             https://www.amazon.com/dp/B09LVSZRZS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 77, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Bamboo Waste Basket | Waste Basket for Bathroo.
    primary_image      https://m.media-amazon.com/images/I/318RY00VlI.
    style                                                            NaN
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "25b2cd7d-983b-431c-8ece-346522a63b2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8af43f45-9d6c-4eb8-9b66-aebf178b9837", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "15e992a2e8ae253860571c9c7b5708f554efa2ff8b015f2b4564e9eb049b00ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1447e1b8-f7c3-41f0-b975-ff80b8b00611", "node_type": "1", "metadata": {}, "hash": "2c98f3d9fdd85d313d27ccc03a962f63841c8f9e3b203cd64a00275e4eea99a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
com/images/I/41sDO1HW2c.
    style                                                            NaN
    material                                                         NaN
    color                                                         Silver
    url                             https://www.amazon.com/dp/B09DGFRM4B
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 75, dtype: object: 'float' object is not iterable
    Error creating embedding for title              itbe Easy Fit Ready-to-Assemble Multipurpose O.
    primary_image      https://m.media-amazon.com/images/I/21NWASZgUV.
    style                                                     Flat Panel
    material                                                 Alloy Steel
    color                                                           Blue
    url                             https://www.amazon.com/dp/B09FR4XSCT
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 76, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Delta ARV18-DN Arvo 18-in Wall Mount Towel Bar.
    primary_image      https://m.media-amazon.com/images/I/11zzs81fXB.
    style                                 18" Towel Bar with 6" Extender
    material                                     Multiple Base Materials
    color                                      Spotshield Brushed Nickel
    url                             https://www.amazon.com/dp/B09LVSZRZS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 77, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Bamboo Waste Basket | Waste Basket for Bathroo.
    primary_image      https://m.media-amazon.com/images/I/318RY00VlI.
    style                                                            NaN
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B08VWTB8CH
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 78, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Way Basics Vinyl Record Storage - 2 Tier Book .
    primary_image      https://m.media-amazon.com/images/I/41YMttt7a5.
    style                                                         Modern
    material                                           Recycled Material
    color                                                          White
    url                             https://www.amazon.com/dp/B075M1PKSW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 79, dtype: object: 'float' object is not iterable
    Error creating embedding for title              TocTen Double Bath Towel Bar - Thicken SUS304 .
    primary_image      https://m.media-amazon.com/images/I/41cFJKXyA5.
    style                                                            NaN
    material                                             Stainless Steel
    color                                                    Matte Black
    url                             https://www.amazon.com/dp/B0BWRVGQRM
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 80, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MoNiBloom Adjustable Bar Stools Set of 2, 360°.
    primary_image      https://m.media-amazon.com/images/I/41jD28iN4b.
    style                                                       Straight
    material                                                         NaN
    color                                                      Dark Grey
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "1447e1b8-f7c3-41f0-b975-ff80b8b00611", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25b2cd7d-983b-431c-8ece-346522a63b2b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "95f94640877bdf245fe2141ea90f89ddc19ab16e9f35f4af85ce4da133475674", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f550c37d-651b-4725-af6e-a59572383cd5", "node_type": "1", "metadata": {}, "hash": "5cc9eb44fb4b534a5f6f4f2893836008701e1c7213a3bb18404d2a90861968a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/318RY00VlI.
    style                                                            NaN
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B08VWTB8CH
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 78, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Way Basics Vinyl Record Storage - 2 Tier Book .
    primary_image      https://m.media-amazon.com/images/I/41YMttt7a5.
    style                                                         Modern
    material                                           Recycled Material
    color                                                          White
    url                             https://www.amazon.com/dp/B075M1PKSW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 79, dtype: object: 'float' object is not iterable
    Error creating embedding for title              TocTen Double Bath Towel Bar - Thicken SUS304 .
    primary_image      https://m.media-amazon.com/images/I/41cFJKXyA5.
    style                                                            NaN
    material                                             Stainless Steel
    color                                                    Matte Black
    url                             https://www.amazon.com/dp/B0BWRVGQRM
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 80, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MoNiBloom Adjustable Bar Stools Set of 2, 360°.
    primary_image      https://m.media-amazon.com/images/I/41jD28iN4b.
    style                                                       Straight
    material                                                         NaN
    color                                                      Dark Grey
    url                             https://www.amazon.com/dp/B0CB7SG59J
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 81, dtype: object: 'float' object is not iterable
    Error creating embedding for title              LANTEFUL Shoe Rack Organizer Shoe Storage Cabi.
    primary_image      https://m.media-amazon.com/images/I/51e8SrHHW3.
    style                                       free standing shoe racks
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C3QDL2XW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 82, dtype: object: 'float' object is not iterable
    Error creating embedding for title              ANDY STAR 24x32 INCH Brushed Nickel Mirror, Ro.
    primary_image      https://m.media-amazon.com/images/I/41MQWfATgg.
    style                                                            NaN
    material                                             Stainless Steel
    color                                                 Brushed Nickel
    url                             https://www.amazon.com/dp/B0CBRGS5D7
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 83, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MJL Furniture Designs Upholstered Cubed/Square.
    primary_image      https://m.media-amazon.com/images/I/410tv-zDYX.
    style                                                   Contemporary
    material                                                        Wood
    color                                                     Smoke Grey
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "f550c37d-651b-4725-af6e-a59572383cd5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1447e1b8-f7c3-41f0-b975-ff80b8b00611", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ea3d6daa2d31b482bb4cf672600eb4233b126f5b79d69cbfff146f4e1965cba9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "254fa32e-0f49-49c6-af37-760303da8830", "node_type": "1", "metadata": {}, "hash": "4891930956b16db50cced2dc69f71870b1c607bfddd88e53a5cffa7d4536d198", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/41jD28iN4b.
    style                                                       Straight
    material                                                         NaN
    color                                                      Dark Grey
    url                             https://www.amazon.com/dp/B0CB7SG59J
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 81, dtype: object: 'float' object is not iterable
    Error creating embedding for title              LANTEFUL Shoe Rack Organizer Shoe Storage Cabi.
    primary_image      https://m.media-amazon.com/images/I/51e8SrHHW3.
    style                                       free standing shoe racks
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C3QDL2XW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 82, dtype: object: 'float' object is not iterable
    Error creating embedding for title              ANDY STAR 24x32 INCH Brushed Nickel Mirror, Ro.
    primary_image      https://m.media-amazon.com/images/I/41MQWfATgg.
    style                                                            NaN
    material                                             Stainless Steel
    color                                                 Brushed Nickel
    url                             https://www.amazon.com/dp/B0CBRGS5D7
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 83, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MJL Furniture Designs Upholstered Cubed/Square.
    primary_image      https://m.media-amazon.com/images/I/410tv-zDYX.
    style                                                   Contemporary
    material                                                        Wood
    color                                                     Smoke Grey
    url                             https://www.amazon.com/dp/B01D378FYE
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 84, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Cpintltr Small Foot Stool Ottoman Modern Accen.
    primary_image      https://m.media-amazon.com/images/I/51CjfUJVuL.
    style                                                            NaN
    material                                                        Pine
    color                                                          Green
    url                             https://www.amazon.com/dp/B0CKPFKDZY
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 85, dtype: object: 'float' object is not iterable
    Error creating embedding for title              YuiHome Extendable Round, Farmhouse 16" Leaf T.
    primary_image      https://m.media-amazon.com/images/I/5175Qzg03L.
    style                                                      Farmhouse
    material                                Rubber Wood, Engineered Wood
    color                                              Natural Wood Wash
    url                             https://www.amazon.com/dp/B0CHVQ6BC5
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 86, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Ergonomic Office Chair,Office Chair, with Lumb.
    primary_image      https://m.media-amazon.com/images/I/51vnoZERmP.
    style                                                      With arms
    material                                                        Foam
    color                                                      All Black
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "254fa32e-0f49-49c6-af37-760303da8830", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f550c37d-651b-4725-af6e-a59572383cd5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7380773aeaa9049334c2af61897520aa457fdba6565f133c1698115b3a52bd36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4168745d-19b8-4165-8576-5f2199c4c174", "node_type": "1", "metadata": {}, "hash": "95e274ffa3b459a7a4d3ce07abb47da74d49283bf7d9a228770b36867185c192", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/410tv-zDYX.
    style                                                   Contemporary
    material                                                        Wood
    color                                                     Smoke Grey
    url                             https://www.amazon.com/dp/B01D378FYE
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 84, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Cpintltr Small Foot Stool Ottoman Modern Accen.
    primary_image      https://m.media-amazon.com/images/I/51CjfUJVuL.
    style                                                            NaN
    material                                                        Pine
    color                                                          Green
    url                             https://www.amazon.com/dp/B0CKPFKDZY
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 85, dtype: object: 'float' object is not iterable
    Error creating embedding for title              YuiHome Extendable Round, Farmhouse 16" Leaf T.
    primary_image      https://m.media-amazon.com/images/I/5175Qzg03L.
    style                                                      Farmhouse
    material                                Rubber Wood, Engineered Wood
    color                                              Natural Wood Wash
    url                             https://www.amazon.com/dp/B0CHVQ6BC5
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 86, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Ergonomic Office Chair,Office Chair, with Lumb.
    primary_image      https://m.media-amazon.com/images/I/51vnoZERmP.
    style                                                      With arms
    material                                                        Foam
    color                                                      All Black
    url                             https://www.amazon.com/dp/B0CBBV4S1P
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 87, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Kate and Laurel Celia Round Metal Foldable Acc.
    primary_image      https://m.media-amazon.com/images/I/31ZMqrgDD8.
    style                                                         Modern
    material                                                        Iron
    color                                                          Black
    url                             https://www.amazon.com/dp/B084WLY61H
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 88, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Lizipai Floating Bedside Table, No Assembly Re.
    primary_image      https://m.media-amazon.com/images/I/41HBX6be98.
    style                                                             no
    material                                                        Wood
    color                                                          White
    url                             https://www.amazon.com/dp/B09NBWCTDS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 89, dtype: object: 'float' object is not iterable
    Error creating embedding for title              CordaRoy's Chenille Bean Bag Ottoman Footstool.
    primary_image      https://m.media-amazon.com/images/I/51HpCirQNA.
    style                                                         Modern
    material                                             Engineered Wood
    color                                                     Rainforest
    url                             https://www.amazon.com/dp/B0BSZ96YG7
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 90,


key: _node_content
string_value:
{"id_": "4168745d-19b8-4165-8576-5f2199c4c174", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "254fa32e-0f49-49c6-af37-760303da8830", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2df5ae3327bec1970e67522f894629e14de2bde4e898001f8c6d745bf4202501", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7897a66a-b7ca-4e55-b829-9289ab094935", "node_type": "1", "metadata": {}, "hash": "a64abcbffbce16ca46b1a17f299eda5384e8d61ad6368f74c2d31e09955a48f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
amazon.com/dp/B0CBBV4S1P
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 87, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Kate and Laurel Celia Round Metal Foldable Acc.
    primary_image      https://m.media-amazon.com/images/I/31ZMqrgDD8.
    style                                                         Modern
    material                                                        Iron
    color                                                          Black
    url                             https://www.amazon.com/dp/B084WLY61H
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 88, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Lizipai Floating Bedside Table, No Assembly Re.
    primary_image      https://m.media-amazon.com/images/I/41HBX6be98.
    style                                                             no
    material                                                        Wood
    color                                                          White
    url                             https://www.amazon.com/dp/B09NBWCTDS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 89, dtype: object: 'float' object is not iterable
    Error creating embedding for title              CordaRoy's Chenille Bean Bag Ottoman Footstool.
    primary_image      https://m.media-amazon.com/images/I/51HpCirQNA.
    style                                                         Modern
    material                                             Engineered Wood
    color                                                     Rainforest
    url                             https://www.amazon.com/dp/B0BSZ96YG7
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 90, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Plebs Home Solid Desktop Store Cart, with Rubb.
    primary_image      https://m.media-amazon.com/images/I/51WFQwBEqj.
    style                                                           Slab
    material                                                        Wood
    color                                                      Dark Blue
    url                             https://www.amazon.com/dp/B0CD7FSWMK
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 91, dtype: object: 'float' object is not iterable
    Error creating embedding for title              ErGear Ergonomic Desk Chair, Office Chair with.
    primary_image      https://m.media-amazon.com/images/I/41C4FUmS-h.
    style                                                      With arms
    material                                                 Memory Foam
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C99D3V15
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 92, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Kingston Brass Millennium Towel-Ring, 7.63", O.
    primary_image      https://m.media-amazon.com/images/I/31+kzwXTjx.
    style                                                            NaN
    material                                                       Brass
    color                                              Oil Rubbed Bronze
    url                             https://www.amazon.com/dp/B00FM0WG7I
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 93, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Homebeez 18.9" Round Velvet Storage Ottoman Mu.
    primary_image      https://m.


key: _node_content
string_value:
{"id_": "7897a66a-b7ca-4e55-b829-9289ab094935", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4168745d-19b8-4165-8576-5f2199c4c174", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f98c53e487005a3df6eb62c4037e2424f2ac0bb62d3e86b42ef8222532cd1659", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90ac4191-e000-45c4-b8eb-9265a68639eb", "node_type": "1", "metadata": {}, "hash": "35a9ff03b0fb650d7aecf6abd72d7db10379ccc6557f94cfae9fd0ec510c9388", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
dtype: object: 'float' object is not iterable
    Error creating embedding for title              Plebs Home Solid Desktop Store Cart, with Rubb.
    primary_image      https://m.media-amazon.com/images/I/51WFQwBEqj.
    style                                                           Slab
    material                                                        Wood
    color                                                      Dark Blue
    url                             https://www.amazon.com/dp/B0CD7FSWMK
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 91, dtype: object: 'float' object is not iterable
    Error creating embedding for title              ErGear Ergonomic Desk Chair, Office Chair with.
    primary_image      https://m.media-amazon.com/images/I/41C4FUmS-h.
    style                                                      With arms
    material                                                 Memory Foam
    color                                                          Black
    url                             https://www.amazon.com/dp/B0C99D3V15
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 92, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Kingston Brass Millennium Towel-Ring, 7.63", O.
    primary_image      https://m.media-amazon.com/images/I/31+kzwXTjx.
    style                                                            NaN
    material                                                       Brass
    color                                              Oil Rubbed Bronze
    url                             https://www.amazon.com/dp/B00FM0WG7I
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 93, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Homebeez 18.9" Round Velvet Storage Ottoman Mu.
    primary_image      https://m.media-amazon.com/images/I/51vTxE-9lH.
    style                                                         Modern
    material                                                        Wood
    color                                                         Orange
    url                             https://www.amazon.com/dp/B09DKG6JDN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 94, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Mickey and Friends Collapsible Nylon Basket Bu.
    primary_image      https://m.media-amazon.com/images/I/410mEc5bbl.
    style                                                            NaN
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B0B7Q5LB2C
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 95, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Homepop Home Decor | Backless Nailhead Trim Co.
    primary_image      https://m.media-amazon.com/images/I/41HPIScA4s.
    style                                                   Contemporary
    material                                                         NaN
    color                                                           Blue
    url                             https://www.amazon.com/dp/B01LWPSVUW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 96, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Camco Life Is Better at The Campsite Outdoor &.
    primary_image      https://m.media-amazon.com/images/I/51DN2is3Zj.


key: _node_content
string_value:
{"id_": "90ac4191-e000-45c4-b8eb-9265a68639eb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7897a66a-b7ca-4e55-b829-9289ab094935", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0ba13b75aef8e1697a7bafad5f1ef3b1843ec5b08938c8e908cc1ed28e650054", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8bc74be9-46d1-46fe-af88-41937cbca43b", "node_type": "1", "metadata": {}, "hash": "b09c28f347f8f1ef4b304e2708a65867e860d98c6f1880dbbcad0785ded4e5cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
dtype: object: 'float' object is not iterable
    Error creating embedding for title              Homebeez 18.9" Round Velvet Storage Ottoman Mu.
    primary_image      https://m.media-amazon.com/images/I/51vTxE-9lH.
    style                                                         Modern
    material                                                        Wood
    color                                                         Orange
    url                             https://www.amazon.com/dp/B09DKG6JDN
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 94, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Mickey and Friends Collapsible Nylon Basket Bu.
    primary_image      https://m.media-amazon.com/images/I/410mEc5bbl.
    style                                                            NaN
    material                                                         NaN
    color                                                            NaN
    url                             https://www.amazon.com/dp/B0B7Q5LB2C
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 95, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Homepop Home Decor | Backless Nailhead Trim Co.
    primary_image      https://m.media-amazon.com/images/I/41HPIScA4s.
    style                                                   Contemporary
    material                                                         NaN
    color                                                           Blue
    url                             https://www.amazon.com/dp/B01LWPSVUW
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 96, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Camco Life Is Better at The Campsite Outdoor &.
    primary_image      https://m.media-amazon.com/images/I/51DN2is3Zj.
    style                                               Outdoor & Indoor
    material                                                      Rubber
    color                                                           Blue
    url                             https://www.amazon.com/dp/B07D7RQNJV
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 97, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MoNiBloom Round Folding Faux Fur Saucer Chair .
    primary_image      https://m.media-amazon.com/images/I/41eoFKL3gK.
    style                                                         Modern
    material                                                   Polyester
    color                                                       Burgundy
    url                             https://www.amazon.com/dp/B0CD7TH3BF
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 98, dtype: object: 'float' object is not iterable
    Error creating embedding for title              YMYNY Vanity Stool Chair with Storage, Square .
    primary_image      https://m.media-amazon.com/images/I/519Am3LPMv.
    style                                                         Modern
    material                                                         NaN
    color                                                     Dusty Blue
    url                             https://www.amazon.com/dp/B0C1NSNDW2
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 99, dtype: object: 'float' object is not iterable
    Error creating embedding for title                   Casual Home 5 Piece Tray Table Set, Espresso
    primary_image      https://m.media-amazon.com/images/I/41WweDJqgZ.
    style                                                 Tray Table Set
    material                                                        Wood
    color                                                       Espresso
    url                             https://www.


key: _node_content
string_value:
{"id_": "8bc74be9-46d1-46fe-af88-41937cbca43b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90ac4191-e000-45c4-b8eb-9265a68639eb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7c1e69899bd969b573c2151c4c29c180a803d57236e5e720c32272232931e6ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b12a9f23-aac7-4477-af12-5d86db8cc30c", "node_type": "1", "metadata": {}, "hash": "2cb41e7463b73051e85943eca0631e3f9fdd68903884799b0d87f2f9ac801a49", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
primary_image      https://m.media-amazon.com/images/I/51DN2is3Zj.
    style                                               Outdoor & Indoor
    material                                                      Rubber
    color                                                           Blue
    url                             https://www.amazon.com/dp/B07D7RQNJV
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 97, dtype: object: 'float' object is not iterable
    Error creating embedding for title              MoNiBloom Round Folding Faux Fur Saucer Chair .
    primary_image      https://m.media-amazon.com/images/I/41eoFKL3gK.
    style                                                         Modern
    material                                                   Polyester
    color                                                       Burgundy
    url                             https://www.amazon.com/dp/B0CD7TH3BF
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 98, dtype: object: 'float' object is not iterable
    Error creating embedding for title              YMYNY Vanity Stool Chair with Storage, Square .
    primary_image      https://m.media-amazon.com/images/I/519Am3LPMv.
    style                                                         Modern
    material                                                         NaN
    color                                                     Dusty Blue
    url                             https://www.amazon.com/dp/B0C1NSNDW2
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 99, dtype: object: 'float' object is not iterable
    Error creating embedding for title                   Casual Home 5 Piece Tray Table Set, Espresso
    primary_image      https://m.media-amazon.com/images/I/41WweDJqgZ.
    style                                                 Tray Table Set
    material                                                        Wood
    color                                                       Espresso
    url                             https://www.amazon.com/dp/B0069H9BYO
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 100, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Simplify Hanging Grey 20-Pocket Shoe Boho Clos.
    primary_image      https://m.media-amazon.com/images/I/41eYiOqsld.
    style                                                            NaN
    material           80% Linen printed nonwoven +20% solid nonwoven.
    color                                                           Grey
    url                             https://www.amazon.com/dp/B09J1RM23P
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 101, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Get Set Style Black Glass Side Table, Square G.
    primary_image      https://m.media-amazon.com/images/I/51gG6ukN1n.
    style                                             Modern and Elegant
    material                                              Tempered Glass
    color                                                    Shiny Black
    url                             https://www.amazon.com/dp/B0C5DH6ZY6
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 102, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Watson & Whitely Swivel Bar Stools Set of 2, F.
    primary_image      https://m.media-amazon.com/images/I/41nDc6aFKo.
    style                                                         Modern
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "b12a9f23-aac7-4477-af12-5d86db8cc30c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8bc74be9-46d1-46fe-af88-41937cbca43b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0b806be0f0b48816b06b2c259ca5117263e005a18d8662bb95feb0b885e707e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3b0d7ded-d7fc-46c7-acae-fce0c99b1cbc", "node_type": "1", "metadata": {}, "hash": "e10b92f5b9762d4212ea27b4fccf33f1f20cdb9fa1cefce22e56bfc1c323c604", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
style                                                 Tray Table Set
    material                                                        Wood
    color                                                       Espresso
    url                             https://www.amazon.com/dp/B0069H9BYO
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 100, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Simplify Hanging Grey 20-Pocket Shoe Boho Clos.
    primary_image      https://m.media-amazon.com/images/I/41eYiOqsld.
    style                                                            NaN
    material           80% Linen printed nonwoven +20% solid nonwoven.
    color                                                           Grey
    url                             https://www.amazon.com/dp/B09J1RM23P
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 101, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Get Set Style Black Glass Side Table, Square G.
    primary_image      https://m.media-amazon.com/images/I/51gG6ukN1n.
    style                                             Modern and Elegant
    material                                              Tempered Glass
    color                                                    Shiny Black
    url                             https://www.amazon.com/dp/B0C5DH6ZY6
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 102, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Watson & Whitely Swivel Bar Stools Set of 2, F.
    primary_image      https://m.media-amazon.com/images/I/41nDc6aFKo.
    style                                                         Modern
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0CKQTTZ5V
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 103, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Sweet Jojo Designs Boho Rainbow Girl Ottoman P.
    primary_image      https://m.media-amazon.com/images/I/31nn4NwuKf.
    style                                                    Shabby Chic
    material                                             Engineered Wood
    color                                                    Multi Color
    url                             https://www.amazon.com/dp/B0BZJYM4Q6
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 104, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Pekokavo Sofa Arm Clip Tray, Side Table for Re.
    primary_image      https://m.media-amazon.com/images/I/51yz-83kj+.
    style                                                         Modern
    material                                                      Bamboo
    color                                                         Bamboo
    url                             https://www.amazon.com/dp/B08SL4GH7G
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 105, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Caroline's Treasures JMA2013HRM2858 Seaweed Sa.
    primary_image      https://m.media-amazon.com/images/I/514qJ5aPtb.
    style                                                         Modern
    material                                                      Rubber
    color                                                   Multicolored
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "3b0d7ded-d7fc-46c7-acae-fce0c99b1cbc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b12a9f23-aac7-4477-af12-5d86db8cc30c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8bfe97b16cf6bf8270cee3edcf35779f36feed944276e288fd480c8259a81acb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "20a08b96-2a96-475d-8d39-8ea73ce36bc7", "node_type": "1", "metadata": {}, "hash": "6013a1b845c369234758ec44631e3c717a51caa6feae054754cef48a75dddb60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/41nDc6aFKo.
    style                                                         Modern
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0CKQTTZ5V
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 103, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Sweet Jojo Designs Boho Rainbow Girl Ottoman P.
    primary_image      https://m.media-amazon.com/images/I/31nn4NwuKf.
    style                                                    Shabby Chic
    material                                             Engineered Wood
    color                                                    Multi Color
    url                             https://www.amazon.com/dp/B0BZJYM4Q6
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 104, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Pekokavo Sofa Arm Clip Tray, Side Table for Re.
    primary_image      https://m.media-amazon.com/images/I/51yz-83kj+.
    style                                                         Modern
    material                                                      Bamboo
    color                                                         Bamboo
    url                             https://www.amazon.com/dp/B08SL4GH7G
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 105, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Caroline's Treasures JMA2013HRM2858 Seaweed Sa.
    primary_image      https://m.media-amazon.com/images/I/514qJ5aPtb.
    style                                                         Modern
    material                                                      Rubber
    color                                                   Multicolored
    url                             https://www.amazon.com/dp/B07SPYM4M5
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 106, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Xchouxer Side Tables Natural Bamboo Sofa Armre.
    primary_image      https://m.media-amazon.com/images/I/511LXRAxI+.
    style                                                         Modern
    material                                                      Bamboo
    color                                                          Beige
    url                             https://www.amazon.com/dp/B08FC5HPBS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 107, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Montessori Learning Toddler Tower, Foldable To.
    primary_image      https://m.media-amazon.com/images/I/51n9ojprZE.
    style                                                         Modern
    material                                                        Wood
    color                                                           Wood
    url                             https://www.amazon.com/dp/B0CKMRJ1H9
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 108, dtype: object: 'float' object is not iterable
    Error creating embedding for title              PAK HOME Set of 2 High Gloss Brown Marble Look.
    primary_image      https://m.media-amazon.com/images/I/51u3oxvEiS.
    style                                                         Tripod
    material                                                        Wood
    color                            Brown Marble High Gloss / Gold Legs
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "20a08b96-2a96-475d-8d39-8ea73ce36bc7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b0d7ded-d7fc-46c7-acae-fce0c99b1cbc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6d1a6d0b2b04c4251e4536b86f2b6821bd6856f259160ffcf89668a34bd8a3f0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff98eace-13e1-4e57-98e6-450b5f2fd90d", "node_type": "1", "metadata": {}, "hash": "335cb37bc0c208b272e7109f58dd0ee4656beeb0ac6286be4d30a7813d730440", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/514qJ5aPtb.
    style                                                         Modern
    material                                                      Rubber
    color                                                   Multicolored
    url                             https://www.amazon.com/dp/B07SPYM4M5
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 106, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Xchouxer Side Tables Natural Bamboo Sofa Armre.
    primary_image      https://m.media-amazon.com/images/I/511LXRAxI+.
    style                                                         Modern
    material                                                      Bamboo
    color                                                          Beige
    url                             https://www.amazon.com/dp/B08FC5HPBS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 107, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Montessori Learning Toddler Tower, Foldable To.
    primary_image      https://m.media-amazon.com/images/I/51n9ojprZE.
    style                                                         Modern
    material                                                        Wood
    color                                                           Wood
    url                             https://www.amazon.com/dp/B0CKMRJ1H9
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 108, dtype: object: 'float' object is not iterable
    Error creating embedding for title              PAK HOME Set of 2 High Gloss Brown Marble Look.
    primary_image      https://m.media-amazon.com/images/I/51u3oxvEiS.
    style                                                         Tripod
    material                                                        Wood
    color                            Brown Marble High Gloss / Gold Legs
    url                             https://www.amazon.com/dp/B09K3MYL91
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 109, dtype: object: 'float' object is not iterable
    Error creating embedding for title              kukli kitchen Spring Door Mat 30 X 17 Inch - S.
    primary_image      https://m.media-amazon.com/images/I/61rRHgR+aE.
    style                                                        Classic
    material                                                      Rubber
    color                                                       Color-33
    url                             https://www.amazon.com/dp/B0BNL8CC5X
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 110, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Dewhut Oversized Pumpkin Couch Accent Chair, M.
    primary_image      https://m.media-amazon.com/images/I/519KoH2aW4.
    style                                                         Modern
    material                                                      Sponge
    color                                                           Navy
    url                             https://www.amazon.com/dp/B0CF8HTCS4
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 111, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Toland Home Garden 800009 Gypsy Garden Flower .
    primary_image      https://m.media-amazon.com/images/I/61gTdPHg5Q.
    style                                               Outdoor & Indoor
    material                                                      Rubber
    color                                                            NaN
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "ff98eace-13e1-4e57-98e6-450b5f2fd90d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "20a08b96-2a96-475d-8d39-8ea73ce36bc7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "62b451c8476b7a64865320344dc7f1f1f9b68897e3aa12356f10b5f65bf5b262", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d1db5c0-66c2-4b14-9a44-33d8c562443e", "node_type": "1", "metadata": {}, "hash": "a2e544184f1c7344f4688f9418ba964d2c8eab44e381e2d095c626d30c7194c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
media-amazon.com/images/I/51u3oxvEiS.
    style                                                         Tripod
    material                                                        Wood
    color                            Brown Marble High Gloss / Gold Legs
    url                             https://www.amazon.com/dp/B09K3MYL91
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 109, dtype: object: 'float' object is not iterable
    Error creating embedding for title              kukli kitchen Spring Door Mat 30 X 17 Inch - S.
    primary_image      https://m.media-amazon.com/images/I/61rRHgR+aE.
    style                                                        Classic
    material                                                      Rubber
    color                                                       Color-33
    url                             https://www.amazon.com/dp/B0BNL8CC5X
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 110, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Dewhut Oversized Pumpkin Couch Accent Chair, M.
    primary_image      https://m.media-amazon.com/images/I/519KoH2aW4.
    style                                                         Modern
    material                                                      Sponge
    color                                                           Navy
    url                             https://www.amazon.com/dp/B0CF8HTCS4
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 111, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Toland Home Garden 800009 Gypsy Garden Flower .
    primary_image      https://m.media-amazon.com/images/I/61gTdPHg5Q.
    style                                               Outdoor & Indoor
    material                                                      Rubber
    color                                                            NaN
    url                             https://www.amazon.com/dp/B00PNJAACG
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 112, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Sintosin Vintage Oval Mirrors for Wall Decor 1.
    primary_image      https://m.media-amazon.com/images/I/41NiOP0+4j.
    style                                                    Shabby Chic
    material                                                        Wood
    color                                                           Oval
    url                             https://www.amazon.com/dp/B0BWJLZF5G
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 113, dtype: object: 'float' object is not iterable
    Error creating embedding for title              BEWISHOME Vanity Stool, Bedroom Vanity Chair w.
    primary_image      https://m.media-amazon.com/images/I/410emoPl2k.
    style                                                         Modern
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0B6FML1VS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 114, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Children's Factory School Age High Back Lounge.
    primary_image      https://m.media-amazon.com/images/I/51ORnRyifR.
    style                                                    Single Seat
    material                                                         NaN
    color                                                       Blue-red
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "0d1db5c0-66c2-4b14-9a44-33d8c562443e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff98eace-13e1-4e57-98e6-450b5f2fd90d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0f926e5ac4dedf97ef797b0d593bd2f66dee20d43cfd12501378d85fa293b5aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74dc9d70-5f1f-454c-bf64-ccd8f450863d", "node_type": "1", "metadata": {}, "hash": "8a08c332c61f44c9b638e2427919b61c020782e42c05fe4c05b66316cf8573ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
primary_image      https://m.media-amazon.com/images/I/61gTdPHg5Q.
    style                                               Outdoor & Indoor
    material                                                      Rubber
    color                                                            NaN
    url                             https://www.amazon.com/dp/B00PNJAACG
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 112, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Sintosin Vintage Oval Mirrors for Wall Decor 1.
    primary_image      https://m.media-amazon.com/images/I/41NiOP0+4j.
    style                                                    Shabby Chic
    material                                                        Wood
    color                                                           Oval
    url                             https://www.amazon.com/dp/B0BWJLZF5G
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 113, dtype: object: 'float' object is not iterable
    Error creating embedding for title              BEWISHOME Vanity Stool, Bedroom Vanity Chair w.
    primary_image      https://m.media-amazon.com/images/I/410emoPl2k.
    style                                                         Modern
    material                                                         NaN
    color                                                          Black
    url                             https://www.amazon.com/dp/B0B6FML1VS
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 114, dtype: object: 'float' object is not iterable
    Error creating embedding for title              Children's Factory School Age High Back Lounge.
    primary_image      https://m.media-amazon.com/images/I/51ORnRyifR.
    style                                                    Single Seat
    material                                                         NaN
    color                                                       Blue-red
    url                             https://www.amazon.com/dp/B00740P05Y
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 115, dtype: object: 'float' object is not iterable
    Error creating embedding for title              FLYJOE Shoe Rack Bench, 3-Tier Freestanding Wo.
    primary_image      https://m.media-amazon.com/images/I/51WQiiIyuS.
    style                                                            NaN
    material                                                         NaN
    color                                                  Rustic Walnut
    url                             https://www.amazon.com/dp/B0CN8NXR1Q
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 116, dtype: object: 'float' object is not iterable
    Error creating embedding for title              FLYZC Counter Height Bar Stools Set of 4, Stoo.
    primary_image      https://m.media-amazon.com/images/I/51jw0SXQMW.
    style                                                       Straight
    material                                                         NaN
    color                                                   Grey & Black
    url                             https://www.amazon.com/dp/B0CH862BV2
    keywords                                                         NaN
    img_description                                                  NaN
    caption                                                          NaN
    Name: 117, dtype: object: 'float' object is not iterable
    Error creating embedding for title              SITMOD Gaming Chairs for Adults with Footrest-.
    primary_image      https://m.media-amazon.com/images/I/41bntfm39U.
    style                                                      With arms
    material                                                 Memory Foam
    color                                                           Grey
    url                             https://www.amazon.


key: _node_content
string_value:
{"id_": "74dc9d70-5f1f-454c-bf64-ccd8f450863d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10ea5ba1-792b-4899-9949-2def50553156", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8a35c83b4ca97a55dbfb4c665f169096f360b8af9befbf8d15dd085af5f31e89", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d1db5c0-66c2-4b14-9a44-33d8c562443e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Tag_caption_images_with_GPT4V.ipynb", "title": "Using GPT-4o mini to tag & caption images", "tokens": 43074, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "136ee9751e78919d54db0543fed2774be6577375365c0fe8151d2db4e077432c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "197deda8-4480-4870-8bc6-c74ff1bfdf3c", "node_type": "1", "metadata": {}, "hash": "9a3229f0e6e16b47df12a3a6929cc1ba2a70609afb7f183e994a98ddd5340d51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


