# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9a7a288e-83b5-44e4-a04d-a31711b79242
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9a7a288e-83b5-44e4-a04d-a31711b79242\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1277ad31-9174-421c-87a8-bd8c49a228d8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c11fd1d4-df32-4a3f-9cea-6149b70abfad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"95dda123d329dc8963a7b011c9ac48603182a2033f96d1bacf44a48be0464c3b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b5b8797-4bbf-43c0-8588-76d72a8a3dde\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4925777faecc5576431ca759e371dfac1edc1e951f5c6a42f16d812accbc3b01\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5931, \"end_char_idx\": 9828, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "ref_doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "chroma:document": "The queries look good, so let's run the searches.\n\n\n\n```python\ndef search_news(\n    query: str,\n    news_api_key: str = news_api_key,\n    num_articles: int = 50,\n    from_datetime: str = \"2023-06-01\",  # the 2023 NBA finals were played in June 2023\n    to_datetime: str = \"2023-06-30\",\n) -> dict:\n    response = requests.get(\n        \"https://newsapi.org/v2/everything\",\n        params={\n            \"q\": query,\n            \"apiKey\": news_api_key,\n            \"pageSize\": num_articles,\n            \"sortBy\": \"relevancy\",\n            \"from\": from_datetime,\n            \"to\": to_datetime,\n        },\n    )\n\n    return response.json()\n\n\narticles = []\n\nfor query in tqdm(queries):\n    result = search_news(query)\n    if result[\"status\"] == \"ok\":\n        articles = articles + result[\"articles\"]\n    else:\n        raise Exception(result[\"message\"])\n\n# remove duplicates\narticles = list({article[\"url\"]: article for article in articles}.values())\n\nprint(\"Total number of articles:\", len(articles))\nprint(\"Top 5 articles of query 1:\", \"\\n\")\n\nfor article in articles[0:5]:\n    print(\"Title:\", article[\"title\"])\n    print(\"Description:\", article[\"description\"])\n    print(\"Content:\", article[\"content\"][0:100] + \"...\")\n    print()\n\n```\n\n\n      0%|          | 0/21 [00:00<?, ?it/s]\n\n\n    Total number of articles: 554\n    Top 5 articles of query 1: \n    \n    Title: Nascar takes on Le Mans as LeBron James gets centenary race under way\n    Description: <ul><li>Nascar has presence at iconic race for first time since 1976</li><li>NBA superstar LeBron James waves flag as honorary starter</li></ul>The crowd chanted \u201cU-S-A! U-S-A!\u201d as Nascar driver lineup for the 24 Hours of Le Mans passed through the city cente\u2026\n    Content: The crowd chanted U-S-A! U-S-A! as Nascar driver lineup for the 24 Hours of Le Mans passed through t...\n    \n    Title: NBA finals predictions: Nuggets or Heat? Our writers share their picks\n    Description: Denver or Miami? Our contributors pick the winner, key players and dark horses before the NBA\u2019s grand finale tips offA lot has been made of the importance of a balanced roster with continuity, but, somehow, still not enough. The Nuggets are the prime example \u2026\n    Content: The Nuggets are here because \n    A lot has been made of the importance of a balanced roster with conti...\n    \n    Title: Unboxing: Michelob ULTRA and Artist Futura Enshrine the NBA Championship In Custom Hand-Painted Bottles\n    Description: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will gather to watch the showdown between the Denver Nuggets and Miami Heat. The beermaker teamed up with artist Futura to remix its newly-designed 2023 Champ Bottle\u2026\n    Content: As the 2022-2023 NBA Championship nears the end, Michelob ULTRA brings joy to sports fans who will g...\n    \n    Title: Futura and Michelob ULTRA Toast to the NBA Finals With Abstract Artwork Crafted From the Brand\u2019s 2023 Limited-Edition Championship Bottles\n    Description: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beermaker is back with its celebratory NBA Champ Bottles. This year, the self-proclaimed MVP of joy is dropping a limited-edition bottle made in collaboration with a\u2026\n    Content: The sun is out to play, and so is Michelob ULTRA. With the 2022-2023 NBA Finals underway, the beerma...\n    \n    Title: Signed and Delivered, Futura and Michelob ULTRA Will Gift Hand-Painted Bottles to This Year\u2019s NBA Championship Team\n    Description: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basketball lovers and sports fans around the globe as the NBA 2022-2023 season comes to a nail-biting close. In collaboration with artist Futura, Michelob ULTRA will\u2026\n    Content: Michelob ULTRA, the MVP of joy and official beer sponsor of the NBA is back to celebrate with basket..."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7b5b8797-4bbf-43c0-8588-76d72a8a3dde
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7b5b8797-4bbf-43c0-8588-76d72a8a3dde\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1277ad31-9174-421c-87a8-bd8c49a228d8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a7a288e-83b5-44e4-a04d-a31711b79242\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bc92a8ac8cc3f2ade2d0ff2bcda308017f9f4142da0520beed75e1b421c8a9c0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dacd88b8-4482-456d-b202-cf59df0b09a6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"943746c439027f798ce77ace1d780f4e946f3f4546acfa84311cbaf7d3eddc71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9836, \"end_char_idx\": 12345, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "ref_doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "chroma:document": "As we can see, oftentimes, the search queries will return a large number of results, many of which are not relevant to the original question asked by the user. In order to improve the quality of the final answer, we use embeddings to re-rank and filter the results.\n\n## 2. Re-rank\n\nDrawing inspiration from [HyDE (Gao et al.)](https://arxiv.org/abs/2212.10496), we first generate a hypothetical ideal answer to rerank our compare our results against. This helps prioritize results that look like good answers, rather than those similar to our question. Here\u2019s the prompt we use to generate our hypothetical answer.\n\n\n\n```python\nHA_INPUT = f\"\"\"\nGenerate a hypothetical answer to the user's question. This answer will be used to rank search results. \nPretend you have all the information you need to answer, but don't use any actual facts. Instead, use placeholders\nlike NAME did something, or NAME said something at PLACE. \n\nUser question: {USER_QUESTION}\n\nFormat: {{\"hypotheticalAnswer\": \"hypothetical answer text\"}}\n\"\"\"\n\nhypothetical_answer = json_gpt(HA_INPUT)[\"hypotheticalAnswer\"]\n\nhypothetical_answer\n\n```\n\n\n\n\n    'The NBA championship was won by TEAM NAME. The MVP was awarded to PLAYER NAME. The last game was held at STADIUM NAME, where both teams played with great energy and enthusiasm. It was a close game, but in the end, TEAM NAME emerged victorious.'\n\n\n\nNow, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.\n\n\n\n```python\nhypothetical_answer_embedding = embeddings(hypothetical_answer)[0]\narticle_embeddings = embeddings(\n    [\n        f\"{article['title']} {article['description']} {article['content'][0:100]}\"\n        for article in articles\n    ]\n)\n\n# Calculate cosine similarity\ncosine_similarities = []\nfor article_embedding in article_embeddings:\n    cosine_similarities.append(dot(hypothetical_answer_embedding, article_embedding))\n\ncosine_similarities[0:10]\n\n```\n\n\n\n\n    [0.7854456526852069,\n     0.8086023500072106,\n     0.8002998147018501,\n     0.7961229569526956,\n     0.798354506673743,\n     0.758216458795653,\n     0.7753754083127359,\n     0.7494958338411927,\n     0.804733946801739,\n     0.8405965885235218]\n\n\n\nFinally, we use these similarity scores to sort and filter the results."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dacd88b8-4482-456d-b202-cf59df0b09a6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dacd88b8-4482-456d-b202-cf59df0b09a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1277ad31-9174-421c-87a8-bd8c49a228d8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b5b8797-4bbf-43c0-8588-76d72a8a3dde\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"aa5960b39569e3dc8cd819d33d63f8ba93f109b8ad6b50aead983d3f5058c697\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1eb60d61-603a-4c5a-9f04-f07981961313\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8928b7b99b2e7d118441c4164cf190081aa3eabe0c5e0f0478aa153ed8c50f87\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10955, \"end_char_idx\": 14906, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "ref_doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "chroma:document": "'The NBA championship was won by TEAM NAME. The MVP was awarded to PLAYER NAME. The last game was held at STADIUM NAME, where both teams played with great energy and enthusiasm. It was a close game, but in the end, TEAM NAME emerged victorious.'\n\n\n\nNow, let's generate embeddings for the search results and the hypothetical answer. We then calculate the cosine distance between these embeddings, giving us a semantic similarity metric. Note that we can simply calculate the dot product in lieu of doing a full cosine similarity calculation since the OpenAI embeddings are returned normalized in our API.\n\n\n\n```python\nhypothetical_answer_embedding = embeddings(hypothetical_answer)[0]\narticle_embeddings = embeddings(\n    [\n        f\"{article['title']} {article['description']} {article['content'][0:100]}\"\n        for article in articles\n    ]\n)\n\n# Calculate cosine similarity\ncosine_similarities = []\nfor article_embedding in article_embeddings:\n    cosine_similarities.append(dot(hypothetical_answer_embedding, article_embedding))\n\ncosine_similarities[0:10]\n\n```\n\n\n\n\n    [0.7854456526852069,\n     0.8086023500072106,\n     0.8002998147018501,\n     0.7961229569526956,\n     0.798354506673743,\n     0.758216458795653,\n     0.7753754083127359,\n     0.7494958338411927,\n     0.804733946801739,\n     0.8405965885235218]\n\n\n\nFinally, we use these similarity scores to sort and filter the results.\n\n\n\n```python\nscored_articles = zip(articles, cosine_similarities)\n\n# Sort articles by cosine similarity\nsorted_articles = sorted(scored_articles, key=lambda x: x[1], reverse=True)\n\n# Print top 5 articles\nprint(\"Top 5 articles:\", \"\\n\")\n\nfor article, score in sorted_articles[0:5]:\n    print(\"Title:\", article[\"title\"])\n    print(\"Description:\", article[\"description\"])\n    print(\"Content:\", article[\"content\"][0:100] + \"...\")\n    print(\"Score:\", score)\n    print()\n\n```\n\n    Top 5 articles: \n    \n    Title: NBA Finals: Denver Nuggets beat Miami Hea, lift thier first-ever NBA title\n    Description: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the NBA Final held on Tuesday at the Ball Arena in Denver\n    Content: Denver Nuggets won their maiden NBA Championship trophy defeating Miami Heat 94-89 in Game 5 of the ...\n    Score: 0.8445817523602124\n    \n    Title: Photos: Denver Nuggets celebrate their first NBA title\n    Description: The Nuggets capped off an impressive postseason by beating the Miami Heat in the NBA Finals.\n    Content: Thousands of supporters watched along the streets of Denver, Colorado as the US National Basketball ...\n    Score: 0.842070667753606\n    \n    Title: Denver Nuggets win first NBA championship title in Game 5 victory over Miami Heat\n    Description: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ball Arena in Denver to take Game 5 of the NBA Finals.\n    Content: The Denver Nuggets won their first NBA championship Monday night, downing the Miami Heat 94-89 at Ba...\n    Score: 0.8409346078172385\n    \n    Title: Denver Nuggets Capture Their First NBA Championship Behind Unbreakable Chemistry\n    Description: After 47 years of waiting, the Denver Nuggets are NBA champions. Led by Nikola Jokic and Jamal Murray, they reached the mountain top by staying true to themselves.\n    Content: DENVER, CO - JUNE 12: Jamal Murray (27) of the Denver Nuggets celebrates as he leaves the court ... ...\n    Score: 0.8405965885235218\n    \n    Title: NBA Finals: Nikola Jokic, Denver Nuggets survive Miami Heat to secure franchise's first NBA championship\n    Description: In a rock-fight of a Game 5, the Denver Nuggets reached the NBA mountaintop from the foothills of the Rockies, winning their first-ever championship and setting Nikola Jokic's legacy as an all-timer in stone.\n    Content: DENVER, COLORADO - JUNE 12: Jamal Murray #27 of the Denver Nuggets reacts during the fourth quarter ...\n    Score: 0.8389716330890262"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1eb60d61-603a-4c5a-9f04-f07981961313
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb", "title": "Question answering using a search API and re-ranking", "tokens": 4009, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1eb60d61-603a-4c5a-9f04-f07981961313\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"1277ad31-9174-421c-87a8-bd8c49a228d8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f4c9e0edabd0a6096b78edba4264943f81a6402b6a4b458c9b133c6f4b48bc\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dacd88b8-4482-456d-b202-cf59df0b09a6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_a_search_API.ipynb\", \"title\": \"Question answering using a search API and re-ranking\", \"tokens\": 4009, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d68ca5d96da23fd657418ebdc2a3e16a37aed47a0705f3ece76fdc87e0809d40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14914, \"end_char_idx\": 17018, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "ref_doc_id": "1277ad31-9174-421c-87a8-bd8c49a228d8", "chroma:document": "Awesome! These results look a lot more relevant to our original query. Now, let's use the top 5 results to generate a final answer.\n\n## 3. Answer\n\n\n\n```python\nformatted_top_results = [\n    {\n        \"title\": article[\"title\"],\n        \"description\": article[\"description\"],\n        \"url\": article[\"url\"],\n    }\n    for article, _score in sorted_articles[0:5]\n]\n\nANSWER_INPUT = f\"\"\"\nGenerate an answer to the user's question based on the given search results. \nTOP_RESULTS: {formatted_top_results}\nUSER_QUESTION: {USER_QUESTION}\n\nInclude as much information as possible in the answer. Reference the relevant search result urls as markdown links.\n\"\"\"\n\ncompletion = client.chat.completions.create(\n    model=GPT_MODEL,\n    messages=[{\"role\": \"user\", \"content\": ANSWER_INPUT}],\n    temperature=0.5,\n    stream=True,\n)\n\ntext = \"\"\nfor chunk in completion:\n    text += chunk.choices[0].delta.content\n    display.clear_output(wait=True)\n    display.display(display.Markdown(text))\n```\n\n\nThe Denver Nuggets won their first-ever NBA championship by defeating the Miami Heat 94-89 in Game 5 of the NBA Finals held on Tuesday at the Ball Arena in Denver, according to this [Business Standard article](https://www.business-standard.com/sports/other-sports-news/nba-finals-denver-nuggets-beat-miami-hea-lift-thier-first-ever-nba-title-123061300285_1.html). Nikola Jokic, the Nuggets' center, was named the NBA Finals MVP. In a rock-fight of a Game 5, the Nuggets reached the NBA mountaintop, securing their franchise's first NBA championship and setting Nikola Jokic's legacy as an all-timer in stone, according to this [Yahoo Sports article](https://sports.yahoo.com/nba-finals-nikola-jokic-denver-nuggets-survive-miami-heat-to-secure-franchises-first-nba-championship-030321214.html). For more information and photos of the Nuggets' celebration, check out this [Al Jazeera article](https://www.aljazeera.com/gallery/2023/6/15/photos-denver-nuggets-celebrate-their-first-nba-title) and this [CNN article](https://www.cnn.com/2023/06/12/sport/denver-nuggets-nba-championship-spt-intl?cid=external-feeds_iluminar_yahoo)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 17236226-3963-416a-93d1-5f65aade71c6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"17236226-3963-416a-93d1-5f65aade71c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9daa6650-83d9-469d-a903-178e3bd488ae\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16a59da98549bc0110606c7cd53af1102ff09372c7b472dfbb3e335e33c89672\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3275, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "# Search reranking with cross-encoders\n\nThis notebook takes you through examples of using a cross-encoder to re-rank search results.\n\nThis is a common use case with our customers, where you've implemented semantic search using embeddings (produced using a [bi-encoder](https://www.sbert.net/examples/applications/retrieve_rerank/README.html#retrieval-bi-encoder)) but the results are not as accurate as your use case requires. A possible cause is that there is some business rule you can use to rerank the documents such as how recent or how popular a document is. \n\nHowever, often there are subtle domain-specific rules that help determine relevancy, and this is where a cross-encoder can be useful. Cross-encoders are more accurate than bi-encoders but they don't scale well, so using them to re-order a shortened list returned by semantic search is the ideal use case.\n\n### Example\n\nConsider a search task with D documents and Q queries.\n\nThe brute force approach of computing every pairwise relevance is expensive; its cost scales as ```D * Q```. This is known as **cross-encoding**.\n\nA faster approach is **embeddings-based search**, in which an embedding is computed once for each document and query, and then re-used multiple times to cheaply compute pairwise relevance. Because embeddings are only computed once, its cost scales as ```D + Q```. This is known as **bi-encoding**.\n\nAlthough embeddings-based search is faster, the quality can be worse. To get the best of both, one common approach is to use embeddings (or another bi-encoder) to cheaply identify top candidates, and then use GPT (or another cross-encoder) to expensively re-rank those top candidates. The cost of this hybrid approach scales as ```(D + Q) * cost of embedding + (N * Q) * cost of re-ranking```, where ```N``` is the number of candidates re-ranked.\n\n### Walkthrough\n\nTo illustrate this approach we'll use ```text-davinci-003``` with ```logprobs``` enabled to build a GPT-powered cross-encoder. Our GPT models have strong general language understanding, which when tuned with some few-shot examples can provide a simple and effective cross-encoding option.\n\nThis notebook drew on this great [article](https://weaviate.io/blog/cross-encoders-as-reranker) by Weaviate, and this [excellent explanation](https://www.sbert.net/examples/applications/cross-encoder/README.html) of bi-encoders vs. cross-encoders from Sentence Transformers.\n\n\n```python\n!pip install openai\n!pip install arxiv\n!pip install tenacity\n!pip install pandas\n!pip install tiktoken\n```\n\n\n```python\nimport arxiv\nfrom math import exp\nimport openai\nimport os\nimport pandas as pd\nfrom tenacity import retry, wait_random_exponential, stop_after_attempt\nimport tiktoken\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\nOPENAI_MODEL = \"gpt-4\"\n```\n\n## Search\n\nWe'll use the arXiv search service for this example, but this step could be performed by any search service you have. The key item to consider is over-fetching slightly to capture all the potentially relevant documents, before re-sorting them.\n\n\n\n```python\nquery = \"how do bi-encoders work for sentence embeddings\"\nsearch = arxiv.Search(\n    query=query, max_results=20, sort_by=arxiv.SortCriterion.Relevance\n)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9daa6650-83d9-469d-a903-178e3bd488ae
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9daa6650-83d9-469d-a903-178e3bd488ae\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"17236226-3963-416a-93d1-5f65aade71c6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6888283326d02001635a0d0efefe7d765f74a37bb7108add97355b3bd22918bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3def4ee-a9fa-495e-960d-4900d0114fd0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3fdb09420e94baba44c98dfab1fd368e4cd8dce2bf4c9e74fce31e2f27fcbb49\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2420, \"end_char_idx\": 5391, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "```python\n!pip install openai\n!pip install arxiv\n!pip install tenacity\n!pip install pandas\n!pip install tiktoken\n```\n\n\n```python\nimport arxiv\nfrom math import exp\nimport openai\nimport os\nimport pandas as pd\nfrom tenacity import retry, wait_random_exponential, stop_after_attempt\nimport tiktoken\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\nOPENAI_MODEL = \"gpt-4\"\n```\n\n## Search\n\nWe'll use the arXiv search service for this example, but this step could be performed by any search service you have. The key item to consider is over-fetching slightly to capture all the potentially relevant documents, before re-sorting them.\n\n\n\n```python\nquery = \"how do bi-encoders work for sentence embeddings\"\nsearch = arxiv.Search(\n    query=query, max_results=20, sort_by=arxiv.SortCriterion.Relevance\n)\n```\n\n\n```python\nresult_list = []\n\nfor result in search.results():\n    result_dict = {}\n\n    result_dict.update({\"title\": result.title})\n    result_dict.update({\"summary\": result.summary})\n\n    # Taking the first url provided\n    result_dict.update({\"article_url\": [x.href for x in result.links][0]})\n    result_dict.update({\"pdf_url\": [x.href for x in result.links][1]})\n    result_list.append(result_dict)\n```\n\n\n```python\nresult_list[0]\n```\n\n\n\n\n    {'title': 'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features',\n     'summary': 'Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n     'article_url': 'http://arxiv.org/abs/2206.07023v2',\n     'pdf_url': 'http://arxiv.org/pdf/2206.07023v2'}"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d3def4ee-a9fa-495e-960d-4900d0114fd0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d3def4ee-a9fa-495e-960d-4900d0114fd0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9daa6650-83d9-469d-a903-178e3bd488ae\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ad925f1a88c40cf8356899a0f5b35f21d50f605d7f98d80ca99db328e9674db9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c5b5f7bc-632c-41f5-b469-a07f82fd1a58\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c69f519d93f62a0553c1392de1d6e130e8779161aea688eb237db3e36ead483f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3685, \"end_char_idx\": 7209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "```python\nresult_list[0]\n```\n\n\n\n\n    {'title': 'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features',\n     'summary': 'Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n     'article_url': 'http://arxiv.org/abs/2206.07023v2',\n     'pdf_url': 'http://arxiv.org/pdf/2206.07023v2'}\n\n\n\n\n```python\nfor i, result in enumerate(result_list):\n    print(f\"{i + 1}: {result['title']}\")\n```\n\n    1: SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features\n    2: Are Classes Clusters?\n    3: Semantic Composition in Visually Grounded Language Models\n    4: Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions\n    5: Learning Probabilistic Sentence Representations from Paraphrases\n    6: Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings\n    7: How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation\n    8: Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences\n    9: Vec2Sent: Probing Sentence Embeddings with Natural Language Generation\n    10: Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings\n    11: SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding\n    12: Learning Joint Representations of Videos and Sentences with Web Image Search\n    13: Character-based Neural Networks for Sentence Pair Modeling\n    14: Train Once, Test Anywhere: Zero-Shot Learning for Text Classification\n    15: Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models\n    16: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models\n    17: In Search for Linear Relations in Sentence Embedding Spaces\n    18: Learning to Borrow -- Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion\n    19: Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences\n    20: Relational Sentence Embedding for Flexible Semantic Matching"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c5b5f7bc-632c-41f5-b469-a07f82fd1a58
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c5b5f7bc-632c-41f5-b469-a07f82fd1a58\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3def4ee-a9fa-495e-960d-4900d0114fd0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6c85f2e494023820356bce1e0d6cc1a2f89019d4b28399c4e1d8691ab132b065\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"728e18ab-01e0-4c55-bc48-9bf03837507f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5551b0d12a201998e13de442ff7e88af66237ff9e6e8c42df97a98906f654f26\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7212, \"end_char_idx\": 10568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "## Cross-encoder\n\nWe'll create a cross-encoder using the ```Completions``` endpoint - the key factors to consider here are:\n- Make your examples domain-specific - the strength of cross-encoders comes when you tailor them to your domain.\n- There is a trade-off between how many potential examples to re-rank vs. processing speed. Consider batching and parallel processing cross-encoder requests to process them more quickly.\n\nThe steps here are:\n- Build a prompt to assess relevance and provide few-shot examples to tune it to your domain.\n- Add a ```logit bias``` for the tokens for ``` Yes``` and ``` No``` to decrease the likelihood of any other tokens occurring.\n- Return the classification of yes/no as well as the ```logprobs```.\n- Rerank the results by the ```logprobs``` keyed on ``` Yes```.\n\n\n```python\ntokens = [\" Yes\", \" No\"]\ntokenizer = tiktoken.encoding_for_model(OPENAI_MODEL)\nids = [tokenizer.encode(token) for token in tokens]\nids[0], ids[1]\n```\n\n\n\n\n    ([3363], [1400])\n\n\n\n\n```python\nprompt = '''\nYou are an Assistant responsible for helping detect whether the retrieved document is relevant to the query. For a given input, you need to output a single token: \"Yes\" or \"No\" indicating the retrieved document is relevant to the query.\n\nQuery: How to plant a tree?\nDocument: \"\"\"Cars were invented in 1886, when German inventor Carl Benz patented his Benz Patent-Motorwagen.[3][4][5] Cars became widely available during the 20th century. One of the first cars affordable by the masses was the 1908 Model T, an American car manufactured by the Ford Motor Company. Cars were rapidly adopted in the US, where they replaced horse-drawn carriages.[6] In Europe and other parts of the world, demand for automobiles did not increase until after World War II.[7] The car is considered an essential part of the developed economy.\"\"\"\nRelevant: No\n\nQuery: Has the coronavirus vaccine been approved?\nDocument: \"\"\"The Pfizer-BioNTech COVID-19 vaccine was approved for emergency use in the United States on December 11, 2020.\"\"\"\nRelevant: Yes\n\nQuery: What is the capital of France?\nDocument: \"\"\"Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Its 19th-century cityscape is crisscrossed by wide boulevards and the River Seine. Beyond such landmarks as the Eiffel Tower and the 12th-century, Gothic Notre-Dame cathedral, the city is known for its cafe culture and designer boutiques along the Rue du Faubourg Saint-Honor\u00e9.\"\"\"\nRelevant: Yes\n\nQuery: What are some papers to learn about PPO reinforcement learning?\nDocument: \"\"\"Proximal Policy Optimization and its Dynamic Version for Sequence Generation: In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance.\"\"\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 728e18ab-01e0-4c55-bc48-9bf03837507f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"728e18ab-01e0-4c55-bc48-9bf03837507f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c5b5f7bc-632c-41f5-b469-a07f82fd1a58\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9e270f76ce0ec33e201d5390232df333e62018df942f0a7333f2fc52e4483516\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7163bd0-b67c-46aa-abe1-6465da808d5d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"409ecad44c74d085069fde984bf6542b2ffdd783ed5867f1d9ab377171fa8448\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8788, \"end_char_idx\": 11847, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "Cars were rapidly adopted in the US, where they replaced horse-drawn carriages.[6] In Europe and other parts of the world, demand for automobiles did not increase until after World War II.[7] The car is considered an essential part of the developed economy.\"\"\"\nRelevant: No\n\nQuery: Has the coronavirus vaccine been approved?\nDocument: \"\"\"The Pfizer-BioNTech COVID-19 vaccine was approved for emergency use in the United States on December 11, 2020.\"\"\"\nRelevant: Yes\n\nQuery: What is the capital of France?\nDocument: \"\"\"Paris, France's capital, is a major European city and a global center for art, fashion, gastronomy and culture. Its 19th-century cityscape is crisscrossed by wide boulevards and the River Seine. Beyond such landmarks as the Eiffel Tower and the 12th-century, Gothic Notre-Dame cathedral, the city is known for its cafe culture and designer boutiques along the Rue du Faubourg Saint-Honor\u00e9.\"\"\"\nRelevant: Yes\n\nQuery: What are some papers to learn about PPO reinforcement learning?\nDocument: \"\"\"Proximal Policy Optimization and its Dynamic Version for Sequence Generation: In sequence generation task, many works use policy gradient for model optimization to tackle the intractable backpropagation issue when maximizing the non-differentiable evaluation metrics or fooling the discriminator in adversarial learning. In this paper, we replace policy gradient with proximal policy optimization (PPO), which is a proved more efficient reinforcement learning algorithm, and propose a dynamic approach for PPO (PPO-dynamic). We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance.\"\"\"\nRelevant: Yes\n\nQuery: Explain sentence embeddings\nDocument: \"\"\"Inside the bubble: exploring the environments of reionisation-era Lyman-\u03b1 emitting galaxies with JADES and FRESCO: We present a study of the environments of 16 Lyman-\u03b1 emitting galaxies (LAEs) in the reionisation era (5.8<z<8) identified by JWST/NIRSpec as part of the JWST Advanced Deep Extragalactic Survey (JADES). Unless situated in sufficiently (re)ionised regions, Lyman-\u03b1 emission from these galaxies would be strongly absorbed by neutral gas in the intergalactic medium (IGM). We conservatively estimate sizes of the ionised regions required to reconcile the relatively low Lyman-\u03b1 velocity offsets (\u0394vLy\u03b1<300kms\u22121) with moderately high Lyman-\u03b1 escape fractions (fesc,Ly\u03b1>5%) observed in our sample of LAEs, indicating the presence of ionised ``bubbles'' with physical sizes of the order of 0.1pMpc\u2272Rion\u22721pMpc in a patchy reionisation scenario where the bubbles are embedded in a fully neutral IGM. Around half of the LAEs in our sample are found to coincide with large-scale galaxy overdensities seen in FRESCO at z\u223c5.8-5.9 and z\u223c7.3, suggesting Lyman-\u03b1 transmission is strongly enhanced in such overdense regions, and underlining the importance of LAEs as tracers of the first large-scale ionised bubbles."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b7163bd0-b67c-46aa-abe1-6465da808d5d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b7163bd0-b67c-46aa-abe1-6465da808d5d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"728e18ab-01e0-4c55-bc48-9bf03837507f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e539bddf44243b0954d8169c567d145a7619af44582a6e6a4f6ae40b004c28fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c5fbe626d8cacb2ca35eb6d8377665387dbf916d1d1093304e5c614c7bc72162\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10323, \"end_char_idx\": 13361, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "We demonstrate the efficacy of PPO and PPO-dynamic on conditional sequence generation tasks including synthetic experiment and chit-chat chatbot. The results show that PPO and PPO-dynamic can beat policy gradient by stability and performance.\"\"\"\nRelevant: Yes\n\nQuery: Explain sentence embeddings\nDocument: \"\"\"Inside the bubble: exploring the environments of reionisation-era Lyman-\u03b1 emitting galaxies with JADES and FRESCO: We present a study of the environments of 16 Lyman-\u03b1 emitting galaxies (LAEs) in the reionisation era (5.8<z<8) identified by JWST/NIRSpec as part of the JWST Advanced Deep Extragalactic Survey (JADES). Unless situated in sufficiently (re)ionised regions, Lyman-\u03b1 emission from these galaxies would be strongly absorbed by neutral gas in the intergalactic medium (IGM). We conservatively estimate sizes of the ionised regions required to reconcile the relatively low Lyman-\u03b1 velocity offsets (\u0394vLy\u03b1<300kms\u22121) with moderately high Lyman-\u03b1 escape fractions (fesc,Ly\u03b1>5%) observed in our sample of LAEs, indicating the presence of ionised ``bubbles'' with physical sizes of the order of 0.1pMpc\u2272Rion\u22721pMpc in a patchy reionisation scenario where the bubbles are embedded in a fully neutral IGM. Around half of the LAEs in our sample are found to coincide with large-scale galaxy overdensities seen in FRESCO at z\u223c5.8-5.9 and z\u223c7.3, suggesting Lyman-\u03b1 transmission is strongly enhanced in such overdense regions, and underlining the importance of LAEs as tracers of the first large-scale ionised bubbles. Considering only spectroscopically confirmed galaxies, we find our sample of UV-faint LAEs (MUV\u2273\u221220mag) and their direct neighbours are generally not able to produce the required ionised regions based on the Lyman-\u03b1 transmission properties, suggesting lower-luminosity sources likely play an important role in carving out these bubbles. These observations demonstrate the combined power of JWST multi-object and slitless spectroscopy in acquiring a unique view of the early stages of Cosmic Reionisation via the most distant LAEs.\"\"\"\nRelevant: No\n\nQuery: {query}\nDocument: \"\"\"{document}\"\"\"\nRelevant:\n'''\n\n\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef document_relevance(query, document):\n    response = openai.chat.completions.create(\n        model=\"text-davinci-003\",\n        message=prompt.format(query=query, document=document),\n        temperature=0,\n        logprobs=True,\n        logit_bias={3363: 1, 1400: 1},\n    )\n\n    return (\n        query,\n        document,\n        response.choices[0].message.content,\n        response.choices[0].logprobs.token_logprobs[0],\n    )\n```\n\n\n```python\ncontent = result_list[0][\"title\"] + \": \" + result_list[0][\"summary\"]\n\n# Set logprobs to 1 so our response will include the most probable token the model identified\nresponse = openai.chat.completions.create(\n    model=OPENAI_MODEL,\n    prompt=prompt.format(query=query, document=content),\n    temperature=0,\n    logprobs=1,\n    logit_bias={3363: 1, 1400: 1},\n    max_tokens=1,\n)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7163bd0-b67c-46aa-abe1-6465da808d5d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f4cde4fe9db3aee0a7d075fa6231802cd792b9625ceb2bf7c5cfe12b001baa5d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49dbaf8e-de1c-42a9-8a2b-c700df543296\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e54dd7e71cf42f3fd9da720f616d855d4058a7f45248ca559200c5d92389c56d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11848, \"end_char_idx\": 14767, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "Considering only spectroscopically confirmed galaxies, we find our sample of UV-faint LAEs (MUV\u2273\u221220mag) and their direct neighbours are generally not able to produce the required ionised regions based on the Lyman-\u03b1 transmission properties, suggesting lower-luminosity sources likely play an important role in carving out these bubbles. These observations demonstrate the combined power of JWST multi-object and slitless spectroscopy in acquiring a unique view of the early stages of Cosmic Reionisation via the most distant LAEs.\"\"\"\nRelevant: No\n\nQuery: {query}\nDocument: \"\"\"{document}\"\"\"\nRelevant:\n'''\n\n\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef document_relevance(query, document):\n    response = openai.chat.completions.create(\n        model=\"text-davinci-003\",\n        message=prompt.format(query=query, document=document),\n        temperature=0,\n        logprobs=True,\n        logit_bias={3363: 1, 1400: 1},\n    )\n\n    return (\n        query,\n        document,\n        response.choices[0].message.content,\n        response.choices[0].logprobs.token_logprobs[0],\n    )\n```\n\n\n```python\ncontent = result_list[0][\"title\"] + \": \" + result_list[0][\"summary\"]\n\n# Set logprobs to 1 so our response will include the most probable token the model identified\nresponse = openai.chat.completions.create(\n    model=OPENAI_MODEL,\n    prompt=prompt.format(query=query, document=content),\n    temperature=0,\n    logprobs=1,\n    logit_bias={3363: 1, 1400: 1},\n    max_tokens=1,\n)\n```\n\n\n```python\nresult = response.choices[0]\nprint(f\"Result was {result.message.content}\")\nprint(f\"Logprobs was {result.logprobs.token_logprobs[0]}\")\nprint(\"\\nBelow is the full logprobs object\\n\\n\")\nprint(result[\"logprobs\"])\n```\n\n    Result was Yes\n    Logprobs was -0.05869877\n    \n    Below is the full logprobs object\n    \n    \n    {\n      \"tokens\": [\n        \"Yes\"\n      ],\n      \"token_logprobs\": [\n        -0.05869877\n      ],\n      \"top_logprobs\": [\n        {\n          \"Yes\": -0.05869877\n        }\n      ],\n      \"text_offset\": [\n        5764\n      ]\n    }\n\n\n\n```python\noutput_list = []\nfor x in result_list:\n    content = x[\"title\"] + \": \" + x[\"summary\"]\n\n    try:\n        output_list.append(document_relevance(query, document=content))\n\n    except Exception as e:\n        print(e)\n```\n\n\n```python\noutput_list[:10]\n```\n\n\n\n\n    [('how do bi-encoders work for sentence embeddings',\n      'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features: Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 49dbaf8e-de1c-42a9-8a2b-c700df543296
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"49dbaf8e-de1c-42a9-8a2b-c700df543296\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa3281cb-2f8c-49ba-9a8b-87c37b9f1b4e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"628ca83a6f66725458c095551d56779191dcb081923d6cc2099251f30c8fede1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95265b2e-db56-427f-a23c-ee3f3f64db13\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69b57bbe8b80f121f619b94de8a976849fd9ddc89d99f2d968ac880bb716bd0a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13364, \"end_char_idx\": 16421, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "```python\nresult = response.choices[0]\nprint(f\"Result was {result.message.content}\")\nprint(f\"Logprobs was {result.logprobs.token_logprobs[0]}\")\nprint(\"\\nBelow is the full logprobs object\\n\\n\")\nprint(result[\"logprobs\"])\n```\n\n    Result was Yes\n    Logprobs was -0.05869877\n    \n    Below is the full logprobs object\n    \n    \n    {\n      \"tokens\": [\n        \"Yes\"\n      ],\n      \"token_logprobs\": [\n        -0.05869877\n      ],\n      \"top_logprobs\": [\n        {\n          \"Yes\": -0.05869877\n        }\n      ],\n      \"text_offset\": [\n        5764\n      ]\n    }\n\n\n\n```python\noutput_list = []\nfor x in result_list:\n    content = x[\"title\"] + \": \" + x[\"summary\"]\n\n    try:\n        output_list.append(document_relevance(query, document=content))\n\n    except Exception as e:\n        print(e)\n```\n\n\n```python\noutput_list[:10]\n```\n\n\n\n\n    [('how do bi-encoders work for sentence embeddings',\n      'SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features: Models based on large-pretrained language models, such as S(entence)BERT,\\nprovide effective and efficient sentence embeddings that show high correlation\\nto human similarity ratings, but lack interpretability. On the other hand,\\ngraph metrics for graph-based meaning representations (e.g., Abstract Meaning\\nRepresentation, AMR) can make explicit the semantic aspects in which two\\nsentences are similar. However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n      'Yes',\n      -0.05326408),\n     ('how do bi-encoders work for sentence embeddings',\n      'Are Classes Clusters?: Sentence embedding models aim to provide general purpose embeddings for\\nsentences. Most of the models studied in this paper claim to perform well on\\nSTS tasks - but they do not report on their suitability for clustering. This\\npaper looks at four recent sentence embedding models (Universal Sentence\\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\\nbrief overview of the ideas behind their implementations."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 95265b2e-db56-427f-a23c-ee3f3f64db13
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"95265b2e-db56-427f-a23c-ee3f3f64db13\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49dbaf8e-de1c-42a9-8a2b-c700df543296\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"711e596a43851da6e09ef3bcb6740ad2f60ab33c3fdc4749922228fe84b4ac75\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a288d6e-1e5e-4a5f-a801-d89693cd6dc5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ec57d9ad7b08d80c894e8827b95c8b79aea64cd57921ff8a288b8515813212bd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14768, \"end_char_idx\": 17738, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "However, such metrics tend to be slow, rely on parsers,\\nand do not reach state-of-the-art performance when rating sentence similarity.\\n  In this work, we aim at the best of both worlds, by learning to induce\\n$S$emantically $S$tructured $S$entence BERT embeddings (S$^3$BERT). Our\\nS$^3$BERT embeddings are composed of explainable sub-embeddings that emphasize\\nvarious semantic sentence features (e.g., semantic roles, negation, or\\nquantification). We show how to i) learn a decomposition of the sentence\\nembeddings into semantic features, through approximation of a suite of\\ninterpretable AMR graph metrics, and how to ii) preserve the overall power of\\nthe neural embeddings by controlling the decomposition learning process with a\\nsecond objective that enforces consistency with the similarity ratings of an\\nSBERT teacher model. In our experimental studies, we show that our approach\\noffers interpretability -- while fully preserving the effectiveness and\\nefficiency of the neural sentence embeddings.',\n      'Yes',\n      -0.05326408),\n     ('how do bi-encoders work for sentence embeddings',\n      'Are Classes Clusters?: Sentence embedding models aim to provide general purpose embeddings for\\nsentences. Most of the models studied in this paper claim to perform well on\\nSTS tasks - but they do not report on their suitability for clustering. This\\npaper looks at four recent sentence embedding models (Universal Sentence\\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\\nbrief overview of the ideas behind their implementations. It then investigates\\nhow well topic classes in two text classification datasets (Amazon Reviews (Ni\\net al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their\\ncorresponding sentence embedding space. While the performance of the resulting\\nclassification model is far from perfect, it is better than random. This is\\ninteresting because the classification model has been constructed in an\\nunsupervised way. The topic classes in these real life topic classification\\ndatasets can be partly reconstructed by clustering the corresponding sentence\\nembeddings.',\n      'No',\n      -0.009535169),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Semantic Composition in Visually Grounded Language Models: What is sentence meaning and its ideal representation? Much of the expressive\\npower of human language derives from semantic composition, the mind's ability\\nto represent meaning hierarchically & relationally over constituents. At the\\nsame time, much sentential meaning is outside the text and requires grounding\\nin sensory, motor, and experiential modalities to be adequately learned.\\nAlthough large language models display considerable compositional ability,\\nrecent work shows that visually-grounded language models drastically fail to\\nrepresent compositional structure."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4a288d6e-1e5e-4a5f-a801-d89693cd6dc5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4a288d6e-1e5e-4a5f-a801-d89693cd6dc5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95265b2e-db56-427f-a23c-ee3f3f64db13\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d43950eada597d280fe2fa812ae77c64dfcab6f37345ed66e1d3d2b2a9fc2c4e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf5010e8448d5116cea8ea74ee9ce14367937d7d88bc31f3ce02ad2060ecd3cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15903, \"end_char_idx\": 19480, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": ": Sentence embedding models aim to provide general purpose embeddings for\\nsentences. Most of the models studied in this paper claim to perform well on\\nSTS tasks - but they do not report on their suitability for clustering. This\\npaper looks at four recent sentence embedding models (Universal Sentence\\nEncoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER\\n(Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a\\nbrief overview of the ideas behind their implementations. It then investigates\\nhow well topic classes in two text classification datasets (Amazon Reviews (Ni\\net al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their\\ncorresponding sentence embedding space. While the performance of the resulting\\nclassification model is far from perfect, it is better than random. This is\\ninteresting because the classification model has been constructed in an\\nunsupervised way. The topic classes in these real life topic classification\\ndatasets can be partly reconstructed by clustering the corresponding sentence\\nembeddings.',\n      'No',\n      -0.009535169),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Semantic Composition in Visually Grounded Language Models: What is sentence meaning and its ideal representation? Much of the expressive\\npower of human language derives from semantic composition, the mind's ability\\nto represent meaning hierarchically & relationally over constituents. At the\\nsame time, much sentential meaning is outside the text and requires grounding\\nin sensory, motor, and experiential modalities to be adequately learned.\\nAlthough large language models display considerable compositional ability,\\nrecent work shows that visually-grounded language models drastically fail to\\nrepresent compositional structure. In this thesis, we explore whether & how\\nmodels compose visually grounded semantics, and how we might improve their\\nability to do so.\\n  Specifically, we introduce 1) WinogroundVQA, a new compositional visual\\nquestion answering benchmark, 2) Syntactic Neural Module Distillation, a\\nmeasure of compositional ability in sentence embedding models, 3) Causal\\nTracing for Image Captioning Models to locate neural representations vital for\\nvision-language composition, 4) Syntactic MeanPool to inject a compositional\\ninductive bias into sentence embeddings, and 5) Cross-modal Attention\\nCongruence Regularization, a self-supervised objective function for\\nvision-language relation alignment. We close by discussing connections of our\\nwork to neuroscience, psycholinguistics, formal semantics, and philosophy.\",\n      'No',\n      -0.008887106),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions: Text embedding models from Natural Language Processing can map text data\\n(e.g. words, sentences, documents) to supposedly meaningful numerical\\nrepresentations (a.k.a. text embeddings). While such models are increasingly\\napplied in social science research, one important issue is often not addressed:\\nthe extent to which these embeddings are valid representations of constructs\\nrelevant for social science research. We therefore propose the use of the\\nclassic construct validity framework to evaluate the validity of text\\nembeddings. We show how this framework can be adapted to the opaque and\\nhigh-dimensional nature of text embeddings, with application to survey\\nquestions. We include several popular text embedding methods (e.g."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a288d6e-1e5e-4a5f-a801-d89693cd6dc5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"20fcad9ddd2da10f51c10cc7c27950361df0fa4874bc420e14752286e0cb487f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b2f1b8ba08f4d4366ce16ecff5edb83fd9c2338dbd04f63975f3f6485c6bf97d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17739, \"end_char_idx\": 21470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "In this thesis, we explore whether & how\\nmodels compose visually grounded semantics, and how we might improve their\\nability to do so.\\n  Specifically, we introduce 1) WinogroundVQA, a new compositional visual\\nquestion answering benchmark, 2) Syntactic Neural Module Distillation, a\\nmeasure of compositional ability in sentence embedding models, 3) Causal\\nTracing for Image Captioning Models to locate neural representations vital for\\nvision-language composition, 4) Syntactic MeanPool to inject a compositional\\ninductive bias into sentence embeddings, and 5) Cross-modal Attention\\nCongruence Regularization, a self-supervised objective function for\\nvision-language relation alignment. We close by discussing connections of our\\nwork to neuroscience, psycholinguistics, formal semantics, and philosophy.\",\n      'No',\n      -0.008887106),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions: Text embedding models from Natural Language Processing can map text data\\n(e.g. words, sentences, documents) to supposedly meaningful numerical\\nrepresentations (a.k.a. text embeddings). While such models are increasingly\\napplied in social science research, one important issue is often not addressed:\\nthe extent to which these embeddings are valid representations of constructs\\nrelevant for social science research. We therefore propose the use of the\\nclassic construct validity framework to evaluate the validity of text\\nembeddings. We show how this framework can be adapted to the opaque and\\nhigh-dimensional nature of text embeddings, with application to survey\\nquestions. We include several popular text embedding methods (e.g. fastText,\\nGloVe, BERT, Sentence-BERT, Universal Sentence Encoder) in our construct\\nvalidity analyses. We find evidence of convergent and discriminant validity in\\nsome cases. We also show that embeddings can be used to predict respondent's\\nanswers to completely new survey questions. Furthermore, BERT-based embedding\\ntechniques and the Universal Sentence Encoder provide more valid\\nrepresentations of survey questions than do others. Our results thus highlight\\nthe necessity to examine the construct validity of text embeddings before\\ndeploying them in social science research.\",\n      'No',\n      -0.008583762),\n     ('how do bi-encoders work for sentence embeddings',\n      'Learning Probabilistic Sentence Representations from Paraphrases: Probabilistic word embeddings have shown effectiveness in capturing notions\\nof generality and entailment, but there is very little work on doing the\\nanalogous type of investigation for sentences. In this paper we define\\nprobabilistic models that produce distributions for sentences. Our\\nbest-performing model treats each word as a linear transformation operator\\napplied to a multivariate Gaussian distribution. We train our models on\\nparaphrases and demonstrate that they naturally capture sentence specificity.\\nWhile our proposed model achieves the best performance overall, we also show\\nthat specificity is represented by simpler architectures via the norm of the\\nsentence vectors. Qualitative analysis shows that our probabilistic model\\ncaptures sentential entailment and provides ways to analyze the specificity and\\npreciseness of individual words.',\n      'No',\n      -0.011975748),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings: Semantic sentence embeddings are usually supervisedly built minimizing\\ndistances between pairs of embeddings of sentences labelled as semantically\\nsimilar by annotators."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed6bd8c9-5bae-4d0a-8e9e-b3c6064defb6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a17e4cabf04e6ce3f62d7743dcf5db38b8c1ad0adff7a0faa38a33a861d00a1e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1f42216-25fc-41b9-b3b5-b0783dcc8df0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78dbfb73c3c8ccf7ea9995913b7e9cd63c473a09dfbf60eb0deacbae86cf9cfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19585, \"end_char_idx\": 23318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "We find evidence of convergent and discriminant validity in\\nsome cases. We also show that embeddings can be used to predict respondent's\\nanswers to completely new survey questions. Furthermore, BERT-based embedding\\ntechniques and the Universal Sentence Encoder provide more valid\\nrepresentations of survey questions than do others. Our results thus highlight\\nthe necessity to examine the construct validity of text embeddings before\\ndeploying them in social science research.\",\n      'No',\n      -0.008583762),\n     ('how do bi-encoders work for sentence embeddings',\n      'Learning Probabilistic Sentence Representations from Paraphrases: Probabilistic word embeddings have shown effectiveness in capturing notions\\nof generality and entailment, but there is very little work on doing the\\nanalogous type of investigation for sentences. In this paper we define\\nprobabilistic models that produce distributions for sentences. Our\\nbest-performing model treats each word as a linear transformation operator\\napplied to a multivariate Gaussian distribution. We train our models on\\nparaphrases and demonstrate that they naturally capture sentence specificity.\\nWhile our proposed model achieves the best performance overall, we also show\\nthat specificity is represented by simpler architectures via the norm of the\\nsentence vectors. Qualitative analysis shows that our probabilistic model\\ncaptures sentential entailment and provides ways to analyze the specificity and\\npreciseness of individual words.',\n      'No',\n      -0.011975748),\n     ('how do bi-encoders work for sentence embeddings',\n      \"Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings: Semantic sentence embeddings are usually supervisedly built minimizing\\ndistances between pairs of embeddings of sentences labelled as semantically\\nsimilar by annotators. Since big labelled datasets are rare, in particular for\\nnon-English languages, and expensive, recent studies focus on unsupervised\\napproaches that require not-paired input sentences. We instead propose a\\nlanguage-independent approach to build large datasets of pairs of informal\\ntexts weakly similar, without manual human effort, exploiting Twitter's\\nintrinsic powerful signals of relatedness: replies and quotes of tweets. We use\\nthe collected pairs to train a Transformer model with triplet-like structures,\\nand we test the generated embeddings on Twitter NLP similarity tasks (PIT and\\nTURL) and STSb. We also introduce four new sentence ranking evaluation\\nbenchmarks of informal texts, carefully extracted from the initial collections\\nof tweets, proving not only that our best model learns classical Semantic\\nTextual Similarity, but also excels on tasks where pairs of sentences are not\\nexact paraphrases. Ablation studies reveal how increasing the corpus size\\ninfluences positively the results, even at 2M samples, suggesting that bigger\\ncollections of Tweets still do not contain redundant information about semantic\\nsimilarities.\",\n      'No',\n      -0.01219046),\n     ('how do bi-encoders work for sentence embeddings',\n      \"How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation: Sentence encoders map sentences to real valued vectors for use in downstream\\napplications. To peek into these representations - e.g., to increase\\ninterpretability of their results - probing tasks have been designed which\\nquery them for linguistic knowledge. However, designing probing tasks for\\nlesser-resourced languages is tricky, because these often lack large-scale\\nannotated data or (high-quality) dependency parsers as a prerequisite of\\nprobing task design in English."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a1f42216-25fc-41b9-b3b5-b0783dcc8df0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a1f42216-25fc-41b9-b3b5-b0783dcc8df0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e0c4ec6-a32f-480a-8cb5-ebb3ae8a961f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cdcec10b099acfc43ab94d954bb17897bd20d149e570c5644f5cd7dbb5c3a85\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e3f285e-a756-406a-8ace-f26d91dfc2d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b03c60e3055708922808458659a58a182a85c9d075f314ee08e216ceb7112b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21471, \"end_char_idx\": 25106, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "Since big labelled datasets are rare, in particular for\\nnon-English languages, and expensive, recent studies focus on unsupervised\\napproaches that require not-paired input sentences. We instead propose a\\nlanguage-independent approach to build large datasets of pairs of informal\\ntexts weakly similar, without manual human effort, exploiting Twitter's\\nintrinsic powerful signals of relatedness: replies and quotes of tweets. We use\\nthe collected pairs to train a Transformer model with triplet-like structures,\\nand we test the generated embeddings on Twitter NLP similarity tasks (PIT and\\nTURL) and STSb. We also introduce four new sentence ranking evaluation\\nbenchmarks of informal texts, carefully extracted from the initial collections\\nof tweets, proving not only that our best model learns classical Semantic\\nTextual Similarity, but also excels on tasks where pairs of sentences are not\\nexact paraphrases. Ablation studies reveal how increasing the corpus size\\ninfluences positively the results, even at 2M samples, suggesting that bigger\\ncollections of Tweets still do not contain redundant information about semantic\\nsimilarities.\",\n      'No',\n      -0.01219046),\n     ('how do bi-encoders work for sentence embeddings',\n      \"How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation: Sentence encoders map sentences to real valued vectors for use in downstream\\napplications. To peek into these representations - e.g., to increase\\ninterpretability of their results - probing tasks have been designed which\\nquery them for linguistic knowledge. However, designing probing tasks for\\nlesser-resourced languages is tricky, because these often lack large-scale\\nannotated data or (high-quality) dependency parsers as a prerequisite of\\nprobing task design in English. To investigate how to probe sentence embeddings\\nin such cases, we investigate sensitivity of probing task results to structural\\ndesign choices, conducting the first such large scale study. We show that\\ndesign choices like size of the annotated probing dataset and type of\\nclassifier used for evaluation do (sometimes substantially) influence probing\\noutcomes. We then probe embeddings in a multilingual setup with design choices\\nthat lie in a 'stable region', as we identify for English, and find that\\nresults on English do not transfer to other languages. Fairer and more\\ncomprehensive sentence-level probing evaluation should thus be carried out on\\nmultiple languages in the future.\",\n      'No',\n      -0.015550519),\n     ('how do bi-encoders work for sentence embeddings',\n      'Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences: Sentence embedding methods offer a powerful approach for working with short\\ntextual constructs or sequences of words. By representing sentences as dense\\nnumerical vectors, many natural language processing (NLP) applications have\\nimproved their performance. However, relatively little is understood about the\\nlatent structure of sentence embeddings. Specifically, research has not\\naddressed whether the length and structure of sentences impact the sentence\\nembedding space and topology. This paper reports research on a set of\\ncomprehensive clustering and network analyses targeting sentence and\\nsub-sentence embedding spaces. Results show that one method generates the most\\nclusterable embeddings. In general, the embeddings of span sub-sentences have\\nbetter clustering properties than the original sentences. The results have\\nimplications for future sentence embedding models and applications."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7e3f285e-a756-406a-8ace-f26d91dfc2d2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7e3f285e-a756-406a-8ace-f26d91dfc2d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1f42216-25fc-41b9-b3b5-b0783dcc8df0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"518cf1bc2985e640bab5da25b367137270d29d069c4cf08a1997359de35db27e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7c7675b-3409-4f87-b7fe-d9208dfe797d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c46a8681ab2e68bb10f4c7f45f736d3412347c95bf944cd65ecc37b6dabca402\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23099, \"end_char_idx\": 26787, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "However, designing probing tasks for\\nlesser-resourced languages is tricky, because these often lack large-scale\\nannotated data or (high-quality) dependency parsers as a prerequisite of\\nprobing task design in English. To investigate how to probe sentence embeddings\\nin such cases, we investigate sensitivity of probing task results to structural\\ndesign choices, conducting the first such large scale study. We show that\\ndesign choices like size of the annotated probing dataset and type of\\nclassifier used for evaluation do (sometimes substantially) influence probing\\noutcomes. We then probe embeddings in a multilingual setup with design choices\\nthat lie in a 'stable region', as we identify for English, and find that\\nresults on English do not transfer to other languages. Fairer and more\\ncomprehensive sentence-level probing evaluation should thus be carried out on\\nmultiple languages in the future.\",\n      'No',\n      -0.015550519),\n     ('how do bi-encoders work for sentence embeddings',\n      'Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences: Sentence embedding methods offer a powerful approach for working with short\\ntextual constructs or sequences of words. By representing sentences as dense\\nnumerical vectors, many natural language processing (NLP) applications have\\nimproved their performance. However, relatively little is understood about the\\nlatent structure of sentence embeddings. Specifically, research has not\\naddressed whether the length and structure of sentences impact the sentence\\nembedding space and topology. This paper reports research on a set of\\ncomprehensive clustering and network analyses targeting sentence and\\nsub-sentence embedding spaces. Results show that one method generates the most\\nclusterable embeddings. In general, the embeddings of span sub-sentences have\\nbetter clustering properties than the original sentences. The results have\\nimplications for future sentence embedding models and applications.',\n      'No',\n      -0.012663184),\n     ('how do bi-encoders work for sentence embeddings',\n      'Vec2Sent: Probing Sentence Embeddings with Natural Language Generation: We introspect black-box sentence embeddings by conditionally generating from\\nthem with the objective to retrieve the underlying discrete sentence. We\\nperceive of this as a new unsupervised probing task and show that it correlates\\nwell with downstream task performance. We also illustrate how the language\\ngenerated from different encoders differs. We apply our approach to generate\\nsentence analogies from sentence embeddings.',\n      'Yes',\n      -0.004863006),\n     ('how do bi-encoders work for sentence embeddings',\n      'Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings: Semantic representation learning for sentences is an important and\\nwell-studied problem in NLP. The current trend for this task involves training\\na Transformer-based sentence encoder through a contrastive objective with text,\\ni.e., clustering sentences with semantically similar meanings and scattering\\nothers. In this work, we find the performance of Transformer models as sentence\\nencoders can be improved by training with multi-modal multi-task losses, using\\nunpaired examples from another modality (e.g., sentences and unrelated\\nimage/audio data). In particular, besides learning by the contrastive loss on\\ntext, our model clusters examples from a non-linguistic domain (e.g.,\\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\\nour framework on unpaired non-linguistic data makes it language-agnostic,\\nenabling it to be widely applicable beyond English NLP."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d7c7675b-3409-4f87-b7fe-d9208dfe797d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d7c7675b-3409-4f87-b7fe-d9208dfe797d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e3f285e-a756-406a-8ace-f26d91dfc2d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d9346aa1132b00a7cae27914f18b5ef8a14d3bb82064664518236431e63ece3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fd4a3bf-16a0-4d17-ac17-eceda84375c6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b77b059a6e4a6df08101daec25b902fbeabe859cb7633f927e152a7795d191ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24908, \"end_char_idx\": 27675, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "In general, the embeddings of span sub-sentences have\\nbetter clustering properties than the original sentences. The results have\\nimplications for future sentence embedding models and applications.',\n      'No',\n      -0.012663184),\n     ('how do bi-encoders work for sentence embeddings',\n      'Vec2Sent: Probing Sentence Embeddings with Natural Language Generation: We introspect black-box sentence embeddings by conditionally generating from\\nthem with the objective to retrieve the underlying discrete sentence. We\\nperceive of this as a new unsupervised probing task and show that it correlates\\nwell with downstream task performance. We also illustrate how the language\\ngenerated from different encoders differs. We apply our approach to generate\\nsentence analogies from sentence embeddings.',\n      'Yes',\n      -0.004863006),\n     ('how do bi-encoders work for sentence embeddings',\n      'Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings: Semantic representation learning for sentences is an important and\\nwell-studied problem in NLP. The current trend for this task involves training\\na Transformer-based sentence encoder through a contrastive objective with text,\\ni.e., clustering sentences with semantically similar meanings and scattering\\nothers. In this work, we find the performance of Transformer models as sentence\\nencoders can be improved by training with multi-modal multi-task losses, using\\nunpaired examples from another modality (e.g., sentences and unrelated\\nimage/audio data). In particular, besides learning by the contrastive loss on\\ntext, our model clusters examples from a non-linguistic domain (e.g.,\\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\\nour framework on unpaired non-linguistic data makes it language-agnostic,\\nenabling it to be widely applicable beyond English NLP. Experiments on 7\\nsemantic textual similarity benchmarks reveal that models trained with the\\nadditional non-linguistic (/images/audio) contrastive objective lead to higher\\nquality sentence embeddings. This indicates that Transformer models are able to\\ngeneralize better by doing a similar task (i.e., clustering) with unpaired\\nexamples from different modalities in a multi-task fashion.',\n      'No',\n      -0.013869206)]\n\n\n\n\n```python\noutput_df = pd.DataFrame(\n    output_list, columns=[\"query\", \"document\", \"prediction\", \"logprobs\"]\n).reset_index()\n# Use exp() to convert logprobs into probability\noutput_df[\"probability\"] = output_df[\"logprobs\"].apply(exp)\n# Reorder based on likelihood of being Yes\noutput_df[\"yes_probability\"] = output_df.apply(\n    lambda x: x[\"probability\"] * -1 + 1\n    if x[\"prediction\"] == \"No\"\n    else x[\"probability\"],\n    axis=1,\n)\noutput_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3fd4a3bf-16a0-4d17-ac17-eceda84375c6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3fd4a3bf-16a0-4d17-ac17-eceda84375c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7c7675b-3409-4f87-b7fe-d9208dfe797d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e11556d4096f38584f323a10b91a9f5d29b45e4cc3f32ff3b4472bd370efe6e0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55647bea4fbc3f70bf955f21af430a006d62dfe440b5afa05380b7569874f93d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25983, \"end_char_idx\": 29606, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "The current trend for this task involves training\\na Transformer-based sentence encoder through a contrastive objective with text,\\ni.e., clustering sentences with semantically similar meanings and scattering\\nothers. In this work, we find the performance of Transformer models as sentence\\nencoders can be improved by training with multi-modal multi-task losses, using\\nunpaired examples from another modality (e.g., sentences and unrelated\\nimage/audio data). In particular, besides learning by the contrastive loss on\\ntext, our model clusters examples from a non-linguistic domain (e.g.,\\nvisual/audio) with a similar contrastive loss at the same time. The reliance of\\nour framework on unpaired non-linguistic data makes it language-agnostic,\\nenabling it to be widely applicable beyond English NLP. Experiments on 7\\nsemantic textual similarity benchmarks reveal that models trained with the\\nadditional non-linguistic (/images/audio) contrastive objective lead to higher\\nquality sentence embeddings. This indicates that Transformer models are able to\\ngeneralize better by doing a similar task (i.e., clustering) with unpaired\\nexamples from different modalities in a multi-task fashion.',\n      'No',\n      -0.013869206)]\n\n\n\n\n```python\noutput_df = pd.DataFrame(\n    output_list, columns=[\"query\", \"document\", \"prediction\", \"logprobs\"]\n).reset_index()\n# Use exp() to convert logprobs into probability\noutput_df[\"probability\"] = output_df[\"logprobs\"].apply(exp)\n# Reorder based on likelihood of being Yes\noutput_df[\"yes_probability\"] = output_df.apply(\n    lambda x: x[\"probability\"] * -1 + 1\n    if x[\"prediction\"] == \"No\"\n    else x[\"probability\"],\n    axis=1,\n)\noutput_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>query</th>\n      <th>document</th>\n      <th>prediction</th>\n      <th>logprobs</th>\n      <th>probability</th>\n      <th>yes_probability</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>SBERT studies Meaning Representations: Decompo...</td>\n      <td>Yes</td>\n      <td>-0.053264</td>\n      <td>0.948130</td>\n      <td>0.948130</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Are Classes Clusters?: Sentence embedding mode...</td>\n      <td>No</td>\n      <td>-0.009535</td>\n      <td>0.990510</td>\n      <td>0.009490</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Semantic Composition in Visually Grounded Lang...</td>\n      <td>No</td>\n      <td>-0.008887</td>\n      <td>0.991152</td>\n      <td>0.008848</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Evaluating the Construct Validity of Text Embe...</td>\n      <td>No</td>\n      <td>-0.008584</td>\n      <td>0.991453</td>\n      <td>0.008547</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Learning Probabilistic Sentence Representation...</td>\n      <td>No</td>\n      <td>-0.011976</td>\n      <td>0.988096</td>\n      <td>0.011904</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3fd4a3bf-16a0-4d17-ac17-eceda84375c6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9bfc8045ce3c3ded0f640cf6cde48ef3c9102e18ecfb65d64a8019361830d21b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec762f01-0783-4631-a185-97e1f1908d57\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4301d325c747ff48cd51bab7eab09876a069081b5833546d979c5590bdb8e8be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "```python\n# Return reranked results\nreranked_df = output_df.sort_values(\n    by=[\"yes_probability\"], ascending=False\n).reset_index()\nreranked_df.head(10)\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>level_0</th>\n      <th>index</th>\n      <th>query</th>\n      <th>document</th>\n      <th>prediction</th>\n      <th>logprobs</th>\n      <th>probability</th>\n      <th>yes_probability</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>16</td>\n      <td>16</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>In Search for Linear Relations in Sentence Emb.</td>\n      <td>Yes</td>\n      <td>-0.004824</td>\n      <td>0.995187</td>\n      <td>0.995187</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>8</td>\n      <td>8</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Vec2Sent: Probing Sentence Embeddings with Nat.</td>\n      <td>Yes</td>\n      <td>-0.004863</td>\n      <td>0.995149</td>\n      <td>0.995149</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>19</td>\n      <td>19</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Relational Sentence Embedding for Flexible Sem.</td>\n      <td>Yes</td>\n      <td>-0.038814</td>\n      <td>0.961930</td>\n      <td>0.961930</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0</td>\n      <td>0</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>SBERT studies Meaning Representations: Decompo.</td>\n      <td>Yes</td>\n      <td>-0.053264</td>\n      <td>0.948130</td>\n      <td>0.948130</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>15</td>\n      <td>15</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Sentence-T5: Scalable Sentence Encoders from P.</td>\n      <td>No</td>\n      <td>-0.291893</td>\n      <td>0.746849</td>\n      <td>0."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ec762f01-0783-4631-a185-97e1f1908d57
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ec762f01-0783-4631-a185-97e1f1908d57\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfef3d8a-5bf4-4c1a-97e3-1c8dcd877aec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"34000819facf129eb5206551ad4e034ce37ef603ec6ede54252dfe6b08b1eec5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ac21f0d2-c294-41b9-a77e-920fec71818d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9ccd774dacbd5341f4ad9452c643df9af4b136c9c2a3c1e202c34a7c516725b8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "</td>\n      <td>Yes</td>\n      <td>-0.004863</td>\n      <td>0.995149</td>\n      <td>0.995149</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>19</td>\n      <td>19</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Relational Sentence Embedding for Flexible Sem.</td>\n      <td>Yes</td>\n      <td>-0.038814</td>\n      <td>0.961930</td>\n      <td>0.961930</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0</td>\n      <td>0</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>SBERT studies Meaning Representations: Decompo.</td>\n      <td>Yes</td>\n      <td>-0.053264</td>\n      <td>0.948130</td>\n      <td>0.948130</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>15</td>\n      <td>15</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Sentence-T5: Scalable Sentence Encoders from P.</td>\n      <td>No</td>\n      <td>-0.291893</td>\n      <td>0.746849</td>\n      <td>0.253151</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>6</td>\n      <td>6</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>How to Probe Sentence Embeddings in Low-Resour.</td>\n      <td>No</td>\n      <td>-0.015551</td>\n      <td>0.984570</td>\n      <td>0.015430</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>18</td>\n      <td>18</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Efficient and Flexible Topic Modeling using Pr.</td>\n      <td>No</td>\n      <td>-0.015296</td>\n      <td>0.984820</td>\n      <td>0.015180</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>9</td>\n      <td>9</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Non-Linguistic Supervision for Contrastive Lea.</td>\n      <td>No</td>\n      <td>-0.013869</td>\n      <td>0.986227</td>\n      <td>0.013773</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>12</td>\n      <td>12</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Character-based Neural Networks for Sentence P.</td>\n      <td>No</td>\n      <td>-0."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ac21f0d2-c294-41b9-a77e-920fec71818d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ac21f0d2-c294-41b9-a77e-920fec71818d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ec762f01-0783-4631-a185-97e1f1908d57\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cad0be18073284f57dea6e5086f0618627b25ca8d19e7e7deefb774874436e29\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3cb5ac0c-7227-4fd7-8452-f32b2150f5b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c4a1cdfad550df49c77588a65fee24326cbbe17b705f2857cb71870de6044e47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "</td>\n      <td>No</td>\n      <td>-0.015551</td>\n      <td>0.984570</td>\n      <td>0.015430</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>18</td>\n      <td>18</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Efficient and Flexible Topic Modeling using Pr.</td>\n      <td>No</td>\n      <td>-0.015296</td>\n      <td>0.984820</td>\n      <td>0.015180</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>9</td>\n      <td>9</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Non-Linguistic Supervision for Contrastive Lea.</td>\n      <td>No</td>\n      <td>-0.013869</td>\n      <td>0.986227</td>\n      <td>0.013773</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>12</td>\n      <td>12</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Character-based Neural Networks for Sentence P.</td>\n      <td>No</td>\n      <td>-0.012866</td>\n      <td>0.987216</td>\n      <td>0.012784</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>7</td>\n      <td>7</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Clustering and Network Analysis for the Embedd.</td>\n      <td>No</td>\n      <td>-0.012663</td>\n      <td>0.987417</td>\n      <td>0.012583</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Inspect our new top document following reranking\nreranked_df[\"document\"][0]\n```\n\n\n\n\n    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\\nrepresentations of sentences. We acquire pairs of very similar sentences\\ndiffering only by a small alterations (such as change of a noun, adding an\\nadjective, noun or punctuation) from datasets for natural language inference\\nusing a simple pattern method. We look into how such a small change within the\\nsentence text affects its representation in the continuous space and how such\\nalterations are reflected by some of the popular sentence embedding models. We\\nfound that vector differences of some embeddings actually reflect small changes\\nwithin a sentence.'"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3cb5ac0c-7227-4fd7-8452-f32b2150f5b5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3cb5ac0c-7227-4fd7-8452-f32b2150f5b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ac21f0d2-c294-41b9-a77e-920fec71818d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"193f23db2dded0ebc66e72d80775caa3b0f70366be282f0bfd6fae4f6bb165de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7d4cb0b-8214-497d-a25c-36e2d81f00ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5e50adc8fdc354796302ff46de056e26161b30d28ff4177475e4807015d7615\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "986227</td>\n      <td>0.013773</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>12</td>\n      <td>12</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Character-based Neural Networks for Sentence P.</td>\n      <td>No</td>\n      <td>-0.012866</td>\n      <td>0.987216</td>\n      <td>0.012784</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>7</td>\n      <td>7</td>\n      <td>how do bi-encoders work for sentence embeddings</td>\n      <td>Clustering and Network Analysis for the Embedd.</td>\n      <td>No</td>\n      <td>-0.012663</td>\n      <td>0.987417</td>\n      <td>0.012583</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Inspect our new top document following reranking\nreranked_df[\"document\"][0]\n```\n\n\n\n\n    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\\nrepresentations of sentences. We acquire pairs of very similar sentences\\ndiffering only by a small alterations (such as change of a noun, adding an\\nadjective, noun or punctuation) from datasets for natural language inference\\nusing a simple pattern method. We look into how such a small change within the\\nsentence text affects its representation in the continuous space and how such\\nalterations are reflected by some of the popular sentence embedding models. We\\nfound that vector differences of some embeddings actually reflect small changes\\nwithin a sentence.'\n\n\n\n## Conclusion\n\nWe've shown how to create a tailored cross-encoder to rerank academic papers. This approach will work best where there are domain-specific nuances that can be used to pick the most relevant corpus for your users, and where some pre-filtering has taken place to limit the amount of data the cross-encoder will need to process. \n\nA few typical use cases we've seen are:\n- Returning a list of 100 most relevant stock reports, then re-ordering into a top 5 or 10 based on the detailed context of a particular set of customer portfolios\n- Running after a classic rules-based search that gets the top 100 or 1000 most relevant results to prune it according to a specific user's context"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a7d4cb0b-8214-497d-a25c-36e2d81f00ab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb", "title": "Search reranking with cross-encoders", "tokens": 8550, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a7d4cb0b-8214-497d-a25c-36e2d81f00ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ad1d1bf9-ca95-40a7-949e-bd42a7444f29\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dced2f13bcff97651066dc82958fcaa9337b1448f97481147a450ebcd8260dd3\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3cb5ac0c-7227-4fd7-8452-f32b2150f5b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Search_reranking_with_cross-encoders.ipynb\", \"title\": \"Search reranking with cross-encoders\", \"tokens\": 8550, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e4ad151a18280c890dd5da569534bb8b198d9f85dbfc5d98023a38124875e86c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33132, \"end_char_idx\": 35818, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "ref_doc_id": "ad1d1bf9-ca95-40a7-949e-bd42a7444f29", "chroma:document": "</td>\n      <td>No</td>\n      <td>-0.012663</td>\n      <td>0.987417</td>\n      <td>0.012583</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Inspect our new top document following reranking\nreranked_df[\"document\"][0]\n```\n\n\n\n\n    'In Search for Linear Relations in Sentence Embedding Spaces: We present an introductory investigation into continuous-space vector\\nrepresentations of sentences. We acquire pairs of very similar sentences\\ndiffering only by a small alterations (such as change of a noun, adding an\\nadjective, noun or punctuation) from datasets for natural language inference\\nusing a simple pattern method. We look into how such a small change within the\\nsentence text affects its representation in the continuous space and how such\\nalterations are reflected by some of the popular sentence embedding models. We\\nfound that vector differences of some embeddings actually reflect small changes\\nwithin a sentence.'\n\n\n\n## Conclusion\n\nWe've shown how to create a tailored cross-encoder to rerank academic papers. This approach will work best where there are domain-specific nuances that can be used to pick the most relevant corpus for your users, and where some pre-filtering has taken place to limit the amount of data the cross-encoder will need to process. \n\nA few typical use cases we've seen are:\n- Returning a list of 100 most relevant stock reports, then re-ordering into a top 5 or 10 based on the detailed context of a particular set of customer portfolios\n- Running after a classic rules-based search that gets the top 100 or 1000 most relevant results to prune it according to a specific user's context\n\n\n### Taking this forward\n\nTaking the few-shot approach, as we have here, can work well when the domain is general enough that a small number of examples will cover most reranking cases. However, as the differences between documents become more specific you may want to consider the ```Fine-tuning``` endpoint to make a more elaborate cross-encoder with a wider variety of examples.\n\nThere is also a latency impact of using ```text-davinci-003``` that you'll need to consider, with even our few examples above taking a couple seconds each - again, the ```Fine-tuning``` endpoint may help you here if you are able to get decent results from an ```ada``` or ```babbage``` fine-tuned model.\n\nWe've used the ```Completions``` endpoint from OpenAI to build our cross-encoder, but this area is well-served by the open-source community. [Here](https://huggingface.co/jeffwan/mmarco-mMiniLMv2-L12-H384-v1) is an example from HuggingFace, for example.\n\nWe hope you find this useful for tuning your search use cases, and look forward to seeing what you build."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: da3e88ec-111f-4abd-b194-88ac4fb52157
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"da3e88ec-111f-4abd-b194-88ac4fb52157\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d1faed8-58e0-47b4-b542-ac24da4fe817\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b8a6f3ffc28b8f5102cba3eae6067a88e995678ef8fdaead23ed51c41fded080\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "## Developing Hallucination Guardrails\n\nA guardrail is a set of rules and checks designed to ensure that the outputs of an LLM are accurate, appropriate, and aligned with user expectations. For more additional information on developing guardrails, you can refer to this [guide on developing guardrails](https://cookbook.openai.com/examples/how_to_use_guardrails).\n\nIn this notebook, we'll walk through the process of developing an output guardrail that specifically checks model outputs for hallucinations. \n\nThis notebook will focus on:\n1. Building out a strong eval set\n2. Identifying specific criteria to measure hallucinations\n3. Improving the accuracy of our guardrail with few-shot prompting\n\n\n\n```python\nfrom concurrent.futures import ThreadPoolExecutor\nfrom IPython.display import display, HTML\nimport json\nimport pandas as pd\nfrom sklearn.metrics import precision_score, recall_score\nfrom typing import List\nfrom openai import OpenAI\n\nclient = OpenAI()\n```\n\n\n```python\n# Function to set up display options for pandas\ndef setup_pandas_display():\n    # Increase display limits\n    pd.set_option('display.max_rows', 500)\n    pd.set_option('display.max_columns', 500)\n\n# Function to make DataFrame scrollable in the notebook output\ndef make_scrollable(df):\n    style = (\n        '<style>'\n        'div.output_scroll {'\n        'resize: both;'\n        'overflow: auto;'\n        '}'\n        '</style>'\n    )\n    html = f\"{style}{df.to_html()}\"\n    display(HTML(html))\n\n# Main function to display DataFrame\ndef display_dataframe(df):\n    setup_pandas_display()    # Enable scrollable view\n    make_scrollable(df)\n\n```\n\n## 1. Building out an eval set\n\nImagine we are a customer support team that is building out an automated support agent. We will be feeding the assistant information from our knowledge base about a specific set of policies for how to handle tickets such as returns, refunds, feedback, and expect the model to follow the policy when interacting with customers.\n\nThe first thing we will do is use GPT-4o to build out a set of policies that we will want to follow.\n\nIf you want to do deep dive into generating synthetic data, you can review our Synthetic Data Generation Cookbook [here](https://cookbook.openai.com/examples/sdg1)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6d1faed8-58e0-47b4-b542-ac24da4fe817
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6d1faed8-58e0-47b4-b542-ac24da4fe817\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da3e88ec-111f-4abd-b194-88ac4fb52157\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a26e31f25bd0e21ae0a3e7458579456580143d2e14e7bc0792e65dbd4e3e3f02\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4578abbe6ecdb750e6c7b47c0d256703667cc897656f106bd8016c7fe80d6e54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 701, \"end_char_idx\": 4395, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\nfrom concurrent.futures import ThreadPoolExecutor\nfrom IPython.display import display, HTML\nimport json\nimport pandas as pd\nfrom sklearn.metrics import precision_score, recall_score\nfrom typing import List\nfrom openai import OpenAI\n\nclient = OpenAI()\n```\n\n\n```python\n# Function to set up display options for pandas\ndef setup_pandas_display():\n    # Increase display limits\n    pd.set_option('display.max_rows', 500)\n    pd.set_option('display.max_columns', 500)\n\n# Function to make DataFrame scrollable in the notebook output\ndef make_scrollable(df):\n    style = (\n        '<style>'\n        'div.output_scroll {'\n        'resize: both;'\n        'overflow: auto;'\n        '}'\n        '</style>'\n    )\n    html = f\"{style}{df.to_html()}\"\n    display(HTML(html))\n\n# Main function to display DataFrame\ndef display_dataframe(df):\n    setup_pandas_display()    # Enable scrollable view\n    make_scrollable(df)\n\n```\n\n## 1. Building out an eval set\n\nImagine we are a customer support team that is building out an automated support agent. We will be feeding the assistant information from our knowledge base about a specific set of policies for how to handle tickets such as returns, refunds, feedback, and expect the model to follow the policy when interacting with customers.\n\nThe first thing we will do is use GPT-4o to build out a set of policies that we will want to follow.\n\nIf you want to do deep dive into generating synthetic data, you can review our Synthetic Data Generation Cookbook [here](https://cookbook.openai.com/examples/sdg1)\n\n\n```python\nsystem_input_prompt = \"\"\"\nYou are a helpful assistant that can generate policies for a support agent at a fictional company to follow. You will be provided with a topic (ie. returns, refunds, feedback) and you are to generate a sample policy for how to handle the it.\n\nWhen constructing the policy, it should contain step-by-step instructions for how to handle the customer inquiry. It should include decision logic for what to do if a customer falls under a certain category, and provide requirements for taking specific actions.\n\"\"\"\n\nuser_policy_example_1 = \"\"\"\"\nRETURN POLICY\n\"\"\"\n\nassistant_policy_example_1 = \"\"\"\nRETURN POLICY\n\n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n\n\"\"\"\n\nuser_policy_input = \"\"\"\n{{POLICY}}\n\"\"\"\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6d1faed8-58e0-47b4-b542-ac24da4fe817\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1bd3cdf139ddbd854e9058a7d2cf6b60ab6604e62bd3f2e77996ce51407510c5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"929580a0-8b53-4fb0-b821-840dfa383685\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3beb5ea763c30e3caf60e5016eb2bbff7e876cae17efc82e16979d4c0e743002\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4398, \"end_char_idx\": 7715, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\ndef generate_policy(policy: str) -> str:\n    input_message = user_policy_input.replace(\"{{POLICY}}\", policy)\n    \n    response = client.chat.completions.create(\n        messages= [\n            {\"role\": \"system\", \"content\": system_input_prompt},\n            {\"role\": \"user\", \"content\": user_policy_example_1},\n            {\"role\": \"assistant\", \"content\": assistant_policy_example_1},\n            {\"role\": \"user\", \"content\": input_message},\n        ],\n        model=\"gpt-4o\"\n    )\n    \n    return response.choices[0].message.content\n\ndef generate_policies() -> List[str]:\n    # List of different types of policies to generate \n    policies = ['PRODUCT FEEDBACK POLICY', 'SHIPPING POLICY', 'WARRANTY POLICY', 'ACCOUNT DELETION', 'COMPLAINT RESOLUTION']\n    \n    with ThreadPoolExecutor() as executor:\n        policy_instructions_list = list(executor.map(generate_policy, policies))\n        \n    return policy_instructions_list\n\npolicy_instructions = generate_policies()\n```\n\nNext we'll take these policies and generate sample customer interactions that do or do not follow the instructions.\n\n\n```python\nsystem_input_prompt = \"\"\"\"\nYou are a helpful assistant that can generate fictional interactions between a support assistant and a customer user. You will be given a set of policy instructions that the support agent is instructed to follow.\n\nBased on the instructions, you must generate a relevant single-turn or multi-turn interaction between the assistant and the user. It should average between 1-3 turns total.\n\nFor a given set of instructions, generate an example conversation that where the assistant either does or does not follow the instructions properly. In the assistant's responses, have it give a combination of single sentence and multi-sentence responses.\n\nThe output must be in a json format with the following three parameters:\n - accurate: \n    - This should be a boolean True or False value that matches whether or not the final assistant message accurately follows the policy instructions\n - kb_article:\n    - This should be the entire policy instruction that is passed in from the user\n - chat_history: \n    - This should contain the entire conversation history except for the final assistant message. \n    - This should be in a format of an array of jsons where each json contains two parameters: role, and content. \n    - Role should be set to either 'user' to represent the customer, or 'assistant' to represent the customer support assistant. \n    - Content should contain the message from the appropriate role.\n    - The final message in the chat history should always come from the user. The assistant response in the following parameter will be a response to this use message.\n - assistant_response: \n    - This should contain the final response from the assistant. This is what we will evaluate to determine whether or not it is accurately following the policy.\n\"\"\"\n\nuser_example_1 = \"\"\"\"\nHere are the policy instructions:\nRETURN POLICY\n\n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 929580a0-8b53-4fb0-b821-840dfa383685
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"929580a0-8b53-4fb0-b821-840dfa383685\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"24d9f77e-c8bf-4a3c-a5d9-b60b9f57a38d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c0a830da3279b08b20aa569ee4c3d32da546d9a95b9cef715fb83b07f8963f5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad05a7c6-5d23-404e-9192-b3a3fef8411d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26edfa70bcb2c4b56abe23aff655525e63558171ae2c415da5643636301ba1f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5749, \"end_char_idx\": 9219, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "Based on the instructions, you must generate a relevant single-turn or multi-turn interaction between the assistant and the user. It should average between 1-3 turns total.\n\nFor a given set of instructions, generate an example conversation that where the assistant either does or does not follow the instructions properly. In the assistant's responses, have it give a combination of single sentence and multi-sentence responses.\n\nThe output must be in a json format with the following three parameters:\n - accurate: \n    - This should be a boolean True or False value that matches whether or not the final assistant message accurately follows the policy instructions\n - kb_article:\n    - This should be the entire policy instruction that is passed in from the user\n - chat_history: \n    - This should contain the entire conversation history except for the final assistant message. \n    - This should be in a format of an array of jsons where each json contains two parameters: role, and content. \n    - Role should be set to either 'user' to represent the customer, or 'assistant' to represent the customer support assistant. \n    - Content should contain the message from the appropriate role.\n    - The final message in the chat history should always come from the user. The assistant response in the following parameter will be a response to this use message.\n - assistant_response: \n    - This should contain the final response from the assistant. This is what we will evaluate to determine whether or not it is accurately following the policy.\n\"\"\"\n\nuser_example_1 = \"\"\"\"\nHere are the policy instructions:\nRETURN POLICY\n\n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n\"\"\"\n\nassistant_example_1 = \"\"\"\n{\n    \"accurate\": \"true\",\n    \"kb_article\": \"1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ad05a7c6-5d23-404e-9192-b3a3fef8411d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ad05a7c6-5d23-404e-9192-b3a3fef8411d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"929580a0-8b53-4fb0-b821-840dfa383685\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d80fe7d800f23f75fab9aefdae054184fd588205119338363c3529440f14338d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"199e0a25-3473-4afe-85e5-8d59f324e75e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b668c2cc8bdba5848c31aad95d8b1964ef1be653197e18250518d23252936e18\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7668, \"end_char_idx\": 10927, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "- unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n\"\"\"\n\nassistant_example_1 = \"\"\"\n{\n    \"accurate\": \"true\",\n    \"kb_article\": \"1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.\",\n    \"chat_history\": [\n        {\n            \"role\": \"user\",\n            \"content\": \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Yes, I am not satisfied with the design\"\n        }\n    ],\n    \"assistant_response\": {\n        \"role\": \"assistant\",\n        \"content\": \"I see. Because the shirt was ordered in the last 30 days, we can provide you with a full refund."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 199e0a25-3473-4afe-85e5-8d59f324e75e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"199e0a25-3473-4afe-85e5-8d59f324e75e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad05a7c6-5d23-404e-9192-b3a3fef8411d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ac004b60a562d7ef8e9a8c3e0adb5a978335b4651ace88f711b73f18be39844a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b186bb309a874c900325b0b981e72c4219c4c26196bca2a9a491b6c4d1d9c89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9220, \"end_char_idx\": 12513, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "**If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.\",\n    \"chat_history\": [\n        {\n            \"role\": \"user\",\n            \"content\": \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Yes, I am not satisfied with the design\"\n        }\n    ],\n    \"assistant_response\": {\n        \"role\": \"assistant\",\n        \"content\": \"I see. Because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?\"\n    }\n}\n\"\"\"\n\nuser_example_2 = \"\"\"\"\nHere are the policy instructions:\nRETURN POLICY\n\n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n\"\"\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"199e0a25-3473-4afe-85e5-8d59f324e75e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9da5485a36e09dbc1fd3f9ac8fee005a8bb0134b5f4c3794dca4f6fef26d0b5c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1cde6bea-e4a6-40e7-9192-3fe587192e66\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"75032821da90bf29599370e3c9942e634d6195eb4670b9f100d635d94b96111a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10838, \"end_char_idx\": 13967, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "Because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?\"\n    }\n}\n\"\"\"\n\nuser_example_2 = \"\"\"\"\nHere are the policy instructions:\nRETURN POLICY\n\n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n\"\"\"\n\nassistant_example_2 = \"\"\"\n{\n    \"accurate\": \"false\",\n    \"kb_article\": \"1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1cde6bea-e4a6-40e7-9192-3fe587192e66
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1cde6bea-e4a6-40e7-9192-3fe587192e66\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e0be3a5-5ab1-4b65-bf5c-6c23a375c53f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"48f17b420aa1ac4304896827ed1c44f556d7e9306aacce59bbf9f9db15f56b77\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"343c2a09-cb42-4afa-b73e-a3134f9c7c08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"309dc015d38a987af30a8a46ebab11f778f8f9f10b960f6ad499390920864a3f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12515, \"end_char_idx\": 15759, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "assistant_example_2 = \"\"\"\n{\n    \"accurate\": \"false\",\n    \"kb_article\": \"1. ** Ask the customer why they want the order replaced ** - Categorize their issue into one of the following buckets: - damaged: They received the product in a damaged state - satisfaction: The customer is not satisfied with the item and does not like the product. - unnecessary: They no longer need the item 2a. **If return category is 'damaged' - Ask customer for a picture of the damaged item - If the item is indeed damaged, continue to step 3 - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund - Skip step 3 and go straight to step 4 2b. **If return category is either 'satisfaction' or 'unnecessary'** - Ask the customer if they can provide feedback on the quality of the item - If the order was made within 30 days, notify them that they are eligible for a full refund - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50% - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund 3. **If the customer is eligible for a return or refund** - Ask the customer to confirm that they would like a return or refund - Once they confirm, process their request 4 **Provide additional support before closing out ticket** - Ask the customer if there is anything else you can do to help them today.\",\n    \"chat_history\": [\n        {\n            \"role\": \"user\",\n            \"content\": \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Yes, I am not satisfied with the design\"\n        }\n    ],\n    \"assistant_response\": {\n        \"role\": \"assistant\",\n        \"content\": \"I see. Because the shirt was ordered in the last 60 days, we cannot process a refund.\"    \n    }\n}\n\"\"\"\n```\n\nNow let's iterate through the policies and generate some examples.\n\n\n```python\ncustomer_interactions = []\n\ndef fetch_response(policy):\n    messages = [\n        { \"role\": \"system\", \"content\": system_input_prompt},\n        { \"role\": \"user\", \"content\": user_example_1},\n        { \"role\": \"assistant\", \"content\": assistant_example_1},\n        { \"role\": \"user\", \"content\": user_example_2},\n        { \"role\": \"assistant\", \"content\": assistant_example_2},\n        { \"role\": \"user\", \"content\": policy}\n    ]\n\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=messages,\n        temperature=0.7,\n        n=10\n    )\n    return response.choices\n\nwith ThreadPoolExecutor() as executor:\n    futures = [executor.submit(fetch_response, policy) for policy in policy_instructions]\n    for future in futures:\n        choices = future.result()\n        customer_interactions.extend([choice.message.content for choice in choices])\n\n```\n\n\n```python\ninteraction_dict = json.loads(customer_interactions[0])\n\ndf_interaction = pd.DataFrame([interaction_dict])\n\n# Pretty print the DataFrame\ndisplay_dataframe(df_interaction)\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 343c2a09-cb42-4afa-b73e-a3134f9c7c08
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"343c2a09-cb42-4afa-b73e-a3134f9c7c08\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1cde6bea-e4a6-40e7-9192-3fe587192e66\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dba06ca708d2f356175f4c43d7c62870a859c40463dfa89486e65ee44c8d0428\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f4795af09643da5c7374ca1bc0fe375d92b2ea8ea151ae1cc9c486a2fc099f75\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14224, \"end_char_idx\": 19157, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Yes, I am not satisfied with the design\"\n        }\n    ],\n    \"assistant_response\": {\n        \"role\": \"assistant\",\n        \"content\": \"I see. Because the shirt was ordered in the last 60 days, we cannot process a refund.\"    \n    }\n}\n\"\"\"\n```\n\nNow let's iterate through the policies and generate some examples.\n\n\n```python\ncustomer_interactions = []\n\ndef fetch_response(policy):\n    messages = [\n        { \"role\": \"system\", \"content\": system_input_prompt},\n        { \"role\": \"user\", \"content\": user_example_1},\n        { \"role\": \"assistant\", \"content\": assistant_example_1},\n        { \"role\": \"user\", \"content\": user_example_2},\n        { \"role\": \"assistant\", \"content\": assistant_example_2},\n        { \"role\": \"user\", \"content\": policy}\n    ]\n\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=messages,\n        temperature=0.7,\n        n=10\n    )\n    return response.choices\n\nwith ThreadPoolExecutor() as executor:\n    futures = [executor.submit(fetch_response, policy) for policy in policy_instructions]\n    for future in futures:\n        choices = future.result()\n        customer_interactions.extend([choice.message.content for choice in choices])\n\n```\n\n\n```python\ninteraction_dict = json.loads(customer_interactions[0])\n\ndf_interaction = pd.DataFrame([interaction_dict])\n\n# Pretty print the DataFrame\ndisplay_dataframe(df_interaction)\n\n```\n\n\n<style>div.output_scroll {resize: both;overflow: auto;}</style><table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>accurate</th>\n      <th>kb_article</th>\n      <th>chat_history</th>\n      <th>assistant_response</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Reception** - Thank the customer for taking the time to provide feedback. - Use a personalized greeting: \"Thank you for your feedback, [Customer Name]. We appreciate your input.\" 2. **Categorize Feedback** - Determine the type of feedback: - **Positive Feedback** - **Negative Feedback** - **Suggestions for Improvement** - Document the feedback under the appropriate category in the internal database. 3. **Responding to Positive Feedback** - Express gratitude: \"We're thrilled to hear that you enjoyed our product. Thank you for letting us know!\" - If possible, offer a small token of appreciation (e.g., discount or voucher for future purchases). 4. **Responding to Negative Feedback** - Apologize sincerely and acknowledge the customer's concerns: \"We apologize that our product did not meet your expectations. Your feedback is important to us.\" - Ask for additional details if necessary to understand the issue better. - Reassure the customer that their feedback will be escalated to the product development team. 5. **Responding to Suggestions** - Acknowledge the suggestion: \"Thank you for your suggestion. We value input from our customers as it helps us improve our products.\" - Inform the customer that their suggestion will be reviewed: \"We will share your idea with our product team for further consideration.\" 6. **Internal Processing** - Log all feedback under the respective category in the internal database. - Forward detailed feedback to the product development team bi-weekly. - High-priority issues should be escalated immediately to the senior management team. 7. **Follow-Up** - Monitor whether the customer's feedback leads to any product updates or changes. - If the customer\u2019s feedback resulted in product enhancement, send a follow-up email to inform them: \"Thank you for your valuable feedback. We wanted to let you know that we've made some improvements based on your input.\" 8. **Closing the Loop** - Ask if there is anything else you can assist the customer with: \"Is there anything else we can help you with today?\" - Close the ticket once all queries and feedback are appropriately addressed. 9. **Continuous Improvement** - Analyze feedback trends monthly to identify recurring issues and areas for improvement. - Use feedback insights for product development meetings and strategic planning sessions. By following these steps, we ensure that customer feedback is valued, documented, and acted upon to continuously improve our product offerings.</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let you know that the new app update is fantastic! The interface is so much smoother now.'}]</td>\n      <td>{'role': 'assistant', 'content': 'Thank you for your feedback! We appreciate your input. We're thrilled to hear that you enjoyed our product. Thank you for letting us know! As a token of our appreciation, we're offering you a 10% discount on your next purchase. Is there anything else we can help you with today?'}</td>\n    </tr>\n  </tbody>\n</table>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"343c2a09-cb42-4afa-b73e-a3134f9c7c08\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"20cc2cd7669c18df8ea9afd7294a83976ce068b9e67770280472c2fa3a2e6640\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78817c7b-9996-4088-b1f2-e46d1d25cff6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"103dc8a23e2ee00b0b14a98c04cecf17389615d68a018cda1f02799e58e4423a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19161, \"end_char_idx\": 21428, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\n# Decode the JSON strings\ndata = [json.loads(entry) for entry in customer_interactions]\n\n# Create a DataFrame from the cleaned data\ndf = pd.DataFrame(data)\n```\n\n\n```python\ndf.head(10)\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>accurate</th>\n      <th>kb_article</th>\n      <th>chat_history</th>\n      <th>assistant_response</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to give...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY\\n\\n1. **Acknowledge Re...</td>\n      <td>[{'role': 'user', 'content': 'I really enjoyed...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to give...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 78817c7b-9996-4088-b1f2-e46d1d25cff6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"78817c7b-9996-4088-b1f2-e46d1d25cff6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d94de7c-2dcd-4bd5-b208-0a3ddc99d42f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e4883933195e372443bd445d0bd13ca9153ba0d30c4c008122e2074b7665eda3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"069ff022-261b-4d49-8d44-583273218e69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93e28ecc8ca9bdaeb2a844e0080f7fb55570b43d03b3eb5399a53613fc1adba8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20425, \"end_char_idx\": 22376, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "**Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to give...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY\\n\\n1. **Acknowledge Re...</td>\n      <td>[{'role': 'user', 'content': 'I really enjoyed...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to give...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I didn't like th...</td>\n      <td>{'role': 'assistant', 'content': 'We apologize...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I have some feed...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I really love th...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>true</td>\n      <td>1. **Acknowledge Reception** - Thank the custo...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 069ff022-261b-4d49-8d44-583273218e69
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"069ff022-261b-4d49-8d44-583273218e69\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78817c7b-9996-4088-b1f2-e46d1d25cff6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2d6e835d2133a001a20bc140cce537293370d783088e01ce57175442a19df98f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ba11be3a-91ba-421d-88d3-840bd82d2521\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3925001a23a585f491d557a768b8a3d7c85968af28c54962ce047131f5d4aba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21429, \"end_char_idx\": 24579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "**Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I didn't like th...</td>\n      <td>{'role': 'assistant', 'content': 'We apologize...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I have some feed...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>true</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I really love th...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>true</td>\n      <td>1. **Acknowledge Reception** - Thank the custo...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n## 2. Constructing our hallucination guardrail\n\nWhen building out our hallucination guardrail, here are some guiding principles:\n\n1. Provide very descriptive metrics to evaluate whether a response is accurate\n- It is important to break down this idea of \"truth\" in easily identifiable metrics that we can measure\n- Metrics like truthfulness and relevance are difficult to measure. Giving concrete ways to score the statement can result in a more accurate guardrail\n2. Ensure consistency across key terminology\n- It is important to keep relevant terms such as knowledge base articles, assistants, and users consistent across the prompt\n- If we begin to use phrases such as assistant vs agent, the model could get confused\n3. Start with the most advanced model\n- There is a cost vs quality trade-off when using the most advanced models. Although GPT-4o may be more expensive, it is important to start with the most advanced model so we can ensure a high degree of accuracy\n- Once we have thoroughly tested out the guardrail and are confident in its performance, we can look to reducing cost by tuning it down to gpt-3.5-turbo\n4. Evaluate each sentence independently and the entire response as a whole\n- If the agent returns a long response, it can be useful to break down the response to individual sentences and evaluate them independently\n- In addition to that, evaluating the whole intent of the message as a whole can ensure that you don't lose important context\n\nWith all of this in mind, let's build out a guardrail system and measure its performance.\n\n\n```python\nguardrail_system_message = \"\"\"You are a highly specialized assistant tasked with reviewing chatbot responses to identify and flag any inaccuracies or hallucinations. For each user message, you must thoroughly analyze the response by considering:\n    1. Knowledge Accuracy: Does the message accurately reflect information found in the knowledge base? Assess not only direct mentions but also contextually inferred knowledge.\n    2. Relevance: Does the message directly address the user's question or statement? Check if the response logically follows the user\u2019s last message, maintaining coherence in the conversation thread.\n    3."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ba11be3a-91ba-421d-88d3-840bd82d2521
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ba11be3a-91ba-421d-88d3-840bd82d2521\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"069ff022-261b-4d49-8d44-583273218e69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"eeca0d1b3d70982cb387260d13c47ca462e7c9c29aa7f7eb05a65f75ceb95eaa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8dd6e262-ccee-4e7e-a010-b0755e92344a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0b57eb644dc0995dfa5a870db5044b3362a64f94d83443dfb13bb21cb0129f33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23938, \"end_char_idx\": 27242, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\nguardrail_system_message = \"\"\"You are a highly specialized assistant tasked with reviewing chatbot responses to identify and flag any inaccuracies or hallucinations. For each user message, you must thoroughly analyze the response by considering:\n    1. Knowledge Accuracy: Does the message accurately reflect information found in the knowledge base? Assess not only direct mentions but also contextually inferred knowledge.\n    2. Relevance: Does the message directly address the user's question or statement? Check if the response logically follows the user\u2019s last message, maintaining coherence in the conversation thread.\n    3. Policy Compliance: Does the message adhere to company policies? Evaluate for subtleties such as misinformation, overpromises, or logical inconsistencies. Ensure the response is polite, non-discriminatory, and practical.\n\nTo perform your task you will be given the following:\n    1. Knowledge Base Articles - These are your source of truth for verifying the content of assistant messages.\n    2. Chat Transcript - Provides context for the conversation between the user and the assistant.\n    3. Assistant Message - The message from the assistant that needs review.\n\nFor each sentence in the assistant's most recent response, assign a score based on the following criteria:\n    1. Factual Accuracy:\n        - Score 1 if the sentence is factually correct and corroborated by the knowledge base.\n        - Score 0 if the sentence contains factual errors or unsubstantiated claims.\n    2. Relevance:\n        - Score 1 if the sentence directly and specifically addresses the user's question or statement without digression.\n        - Score 0 if the sentence is tangential or does not build logically on the conversation thread.\n    3. Policy Compliance:\n        - Score 1 if the response complies with all company policies including accuracy, ethical guidelines, and user engagement standards.\n        - Score 0 if it violates any aspect of the policies, such as misinformation or inappropriate content.\n    4. Contextual Coherence:\n        - Score 1 if the sentence maintains or enhances the coherence of the conversation, connecting logically with preceding messages.\n        - Score 0 if it disrupts the flow or context of the conversation.\n\nInclude in your response an array of JSON objects for each evaluated sentence. Each JSON object should contain:\n    - `sentence`: Text of the evaluated sentence.\n    - `factualAccuracy`: Score for factual correctness (0 or 1).\n    - `factualReference`: If scored 1, cite the exact line(s) from the knowledge base. If scored 0, provide a rationale.\n    - `relevance`: Score for relevance to the user\u2019s question (0 or 1).\n    - `policyCompliance`: Score for adherence to company policies (0 or 1).\n    - `contextualCoherence`: Score for maintaining conversation coherence (0 or 1).\n\nALWAYS RETURN YOUR RESPONSE AS AN ARRAY OF JSONS.\n\"\"\"\n\nfs_user_1 = \"\"\"\n\n## Knowledge Base Articles: \n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8dd6e262-ccee-4e7e-a010-b0755e92344a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8dd6e262-ccee-4e7e-a010-b0755e92344a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba11be3a-91ba-421d-88d3-840bd82d2521\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"73e7450752e7c24c629a0fda1734c759a0ffbda55e89a35232b9deff4a59309e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a03eb692-d487-4ddb-aed5-564be422c37a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6705a766f37c8f979827d1fa18f56ce17febe76016151668b5ce7c4683e99d74\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25461, \"end_char_idx\": 28698, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "2. Relevance:\n        - Score 1 if the sentence directly and specifically addresses the user's question or statement without digression.\n        - Score 0 if the sentence is tangential or does not build logically on the conversation thread.\n    3. Policy Compliance:\n        - Score 1 if the response complies with all company policies including accuracy, ethical guidelines, and user engagement standards.\n        - Score 0 if it violates any aspect of the policies, such as misinformation or inappropriate content.\n    4. Contextual Coherence:\n        - Score 1 if the sentence maintains or enhances the coherence of the conversation, connecting logically with preceding messages.\n        - Score 0 if it disrupts the flow or context of the conversation.\n\nInclude in your response an array of JSON objects for each evaluated sentence. Each JSON object should contain:\n    - `sentence`: Text of the evaluated sentence.\n    - `factualAccuracy`: Score for factual correctness (0 or 1).\n    - `factualReference`: If scored 1, cite the exact line(s) from the knowledge base. If scored 0, provide a rationale.\n    - `relevance`: Score for relevance to the user\u2019s question (0 or 1).\n    - `policyCompliance`: Score for adherence to company policies (0 or 1).\n    - `contextualCoherence`: Score for maintaining conversation coherence (0 or 1).\n\nALWAYS RETURN YOUR RESPONSE AS AN ARRAY OF JSONS.\n\"\"\"\n\nfs_user_1 = \"\"\"\n\n## Knowledge Base Articles: \n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n    \n## Chat Transcript:\n    [\n        {\n            \"role\": \"user\",\n            \"content: \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a03eb692-d487-4ddb-aed5-564be422c37a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a03eb692-d487-4ddb-aed5-564be422c37a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8dd6e262-ccee-4e7e-a010-b0755e92344a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8d5c73a87b943441c3d7f86e1605693ab712399dabd6095da112a9ef4dc35cc7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cbeaead1da77ffdbb0b18d6c5a149cd4f6fcc3b7af1f2f87418bc32ed59ca747\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27195, \"end_char_idx\": 30097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "- unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n    \n## Chat Transcript:\n    [\n        {\n            \"role\": \"user\",\n            \"content: \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content: \"Yes, I am not satisfied with the design\"\n        }\n    ]\n\n## Assistant Message:\nI see, because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?\n\"\"\"\n\nfs_assistant_1 = \"\"\"[\n    {\n        \"sentence\": \"I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund.\",\n        \"factualAccuracy\": 1,\n        \"factualReference\": \"If the order was made within 30 days, notify them that they are eligible for a full refund\",\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    },\n    {\n        \"sentence\": \"Would you like me to process the refund?\",\n        \"factualAccuracy\": 1,\n        \"factualReference\": \"If the order was made within 30 days, notify them that they are eligible for a full refund\",\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    }\n]\n\"\"\"\nfs_user_2 = \"\"\"\n## Knowledge Base Articles: \n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a03eb692-d487-4ddb-aed5-564be422c37a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ca70ee09f501ddb0beed6bcd955d4c95f6d76a5adc65abef251ab0280573cb2f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f8a9097c-67b2-42b7-afb4-15b969e07443\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cd6d1a0066a4fc01c0624405a08c348df7eb89d80b4087e43037c977023f9ee6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28619, \"end_char_idx\": 31831, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content: \"Yes, I am not satisfied with the design\"\n        }\n    ]\n\n## Assistant Message:\nI see, because the shirt was ordered in the last 30 days, we can provide you with a full refund. Would you like me to process the refund?\n\"\"\"\n\nfs_assistant_1 = \"\"\"[\n    {\n        \"sentence\": \"I see, because the shirt was ordered in the last 30 days, we can provide you with a full refund.\",\n        \"factualAccuracy\": 1,\n        \"factualReference\": \"If the order was made within 30 days, notify them that they are eligible for a full refund\",\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    },\n    {\n        \"sentence\": \"Would you like me to process the refund?\",\n        \"factualAccuracy\": 1,\n        \"factualReference\": \"If the order was made within 30 days, notify them that they are eligible for a full refund\",\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    }\n]\n\"\"\"\nfs_user_2 = \"\"\"\n## Knowledge Base Articles: \n1. ** Ask the customer why they want the order replaced **\n    - Categorize their issue into one of the following buckets:\n        - damaged: They received the product in a damaged state\n        - satisfaction: The customer is not satisfied with the item and does not like the product.\n        - unnecessary: They no longer need the item\n2a. **If return category is 'damaged'\n    - Ask customer for a picture of the damaged item\n    - If the item is indeed damaged, continue to step 3\n    - If the item is not damaged, notify the customer that this does not meet our requirements for return and they are not eligible for a refund\n    - Skip step 3 and go straight to step 4\n\n2b. **If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n    \n## Chat Transcript:\n    [\n        {\n            \"role\": \"user\",\n            \"content: \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content: \"Yes, I am not satisfied with the design\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"I see, because the shirt was ordered in the last 60 days, we cannot process a refund.\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f8a9097c-67b2-42b7-afb4-15b969e07443
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f8a9097c-67b2-42b7-afb4-15b969e07443\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1b5d5e9e-baf6-4a72-9565-1cac07f9d8fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"68094477cff3e2264ba25dbdb507ea4b5bc11b9037bd7ce79d24d4b943c681db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da8fcea9-71b1-49ee-a3a6-a84d416b6f73\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ca31136485cd21ed9dee3b71622a92b1fdde6f25f29d4aa87b4a8c82e505d3a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30435, \"end_char_idx\": 32471, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "**If return category is either 'satisfaction' or 'unnecessary'**\n    - Ask the customer if they can provide feedback on the quality of the item\n    - If the order was made within 30 days, notify them that they are eligible for a full refund\n    - If the order was made within 31-60 days, notify them that they are eligible for a partial refund of 50%\n    - If the order was made greater than 60 days ago, notify them that they are not eligible for a refund\n\n3. **If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n    \n## Chat Transcript:\n    [\n        {\n            \"role\": \"user\",\n            \"content: \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content: \"Yes, I am not satisfied with the design\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"I see, because the shirt was ordered in the last 60 days, we cannot process a refund.\"\n        }\n        ]\n## Assistant Message: \nI see, because the shirt was ordered in the last 60 days, we cannot process a refund.\n\"\"\"\n\nfs_assistant_2 = \"\"\"'[\n    {\n        \"sentence\": \"I see, because the shirt was ordered in the last 60 days, we cannot process a refund.\",\n        \"factualAccuracy\": 0,\n        \"knowledgeReference: \"If an order was placed within 60 days, you must process a partial refund.\"\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    }\n]\"\"\"\n\n\nuser_input = \"\"\"\n## Knowledge Base Articles\n{kb_articles}\n\n## Chat Transcript\n{transcript}\n\n## Assistant Message:\n{message}\n\"\"\"\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: da8fcea9-71b1-49ee-a3a6-a84d416b6f73
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"da8fcea9-71b1-49ee-a3a6-a84d416b6f73\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f8a9097c-67b2-42b7-afb4-15b969e07443\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0cfdc715bda8b2f7b1853539ff2adaa1794a1ecda89219b0baf290f06ee4473b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8332cc8f-925c-43a2-8abf-ffa88912630a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"62ee1fa69eb8e9e68ed0d17122304396a468f519f8614cbf4feb83bae0ea6f02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30896, \"end_char_idx\": 34646, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "**If the customer is eligible for a return or refund**\n    - Ask the customer to confirm that they would like a return or refund\n    - Once they confirm, process their request\n\n4 **Provide additional support before closing out ticket**\n    - Ask the customer if there is anything else you can do to help them today.\n    \n## Chat Transcript:\n    [\n        {\n            \"role\": \"user\",\n            \"content: \"I would like to return this shirt\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Hi there, I'm happy to help with processing this return. Can you please provide an explanation for why you'd like to return this shirt?\"\n        },\n        {\n            \"role\": \"user\",\n            \"content: \"Yes, I am not satisfied with the design\"\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"I see, because the shirt was ordered in the last 60 days, we cannot process a refund.\"\n        }\n        ]\n## Assistant Message: \nI see, because the shirt was ordered in the last 60 days, we cannot process a refund.\n\"\"\"\n\nfs_assistant_2 = \"\"\"'[\n    {\n        \"sentence\": \"I see, because the shirt was ordered in the last 60 days, we cannot process a refund.\",\n        \"factualAccuracy\": 0,\n        \"knowledgeReference: \"If an order was placed within 60 days, you must process a partial refund.\"\n        \"relevance\": 1,\n        \"policyCompliance\": 1,\n        \"contextualCoherence\": 1\n    }\n]\"\"\"\n\n\nuser_input = \"\"\"\n## Knowledge Base Articles\n{kb_articles}\n\n## Chat Transcript\n{transcript}\n\n## Assistant Message:\n{message}\n\"\"\"\n```\n\n\n```python\nhallucination_outputs = []\n\ndef validate_hallucinations(row):\n    kb_articles = row['kb_article']\n    chat_history = row['chat_history']\n    assistant_response = row['assistant_response']\n    \n    user_input_filled = user_input.format(\n        kb_articles=kb_articles,\n        transcript=chat_history,\n        message=assistant_response\n    )\n    \n    messages = [\n        { \"role\": \"system\", \"content\": guardrail_system_message},\n        { \"role\": \"user\", \"content\": fs_user_1},\n        { \"role\": \"assistant\", \"content\": fs_assistant_1},\n        { \"role\": \"user\", \"content\": fs_user_2},\n        { \"role\": \"assistant\", \"content\": fs_assistant_2},\n        { \"role\": \"user\", \"content\": user_input_filled}\n    ]\n\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=messages,\n        temperature=0.7,\n        n=10\n    )\n    return response.choices\n\n# Create an empty list to store the results\nresults_list = []\n\ndef process_row(row):\n    choices = validate_hallucinations(row)\n    response_json = choices[0].message.content \n    # Parse the response content as JSON\n    response_data = json.loads(response_json)\n    \n    for response_item in response_data:\n        # Sum up the scores of the properties\n        score_sum = (\n            response_item.get('factualAccuracy', 0) +\n            response_item.get('relevance', 0) +\n            response_item.get('policyCompliance', 0) +\n            response_item.get('contextualCoherence', 0)\n        )\n        \n        # Determine if the response item is a pass or fail\n        hallucination_status = 'Pass' if score_sum == 4 else 'Fail'\n        \n        results_list.append({\n            'accurate': row['accurate'],\n            'hallucination': hallucination_status,\n            'kb_article': row['kb_article'],\n            'chat_history': row['chat_history'],\n            'assistant_response': row['assistant_response']\n        })\n\n# Use ThreadPoolExecutor to parallelize the processing of rows\nwith ThreadPoolExecutor() as executor:\n    executor.map(process_row, [row for index, row in df.iterrows()])\n\n# Convert the list to a DataFrame\nresults_df = pd.DataFrame(results_list)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8332cc8f-925c-43a2-8abf-ffa88912630a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8332cc8f-925c-43a2-8abf-ffa88912630a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da8fcea9-71b1-49ee-a3a6-a84d416b6f73\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"86c0718e5516c24477144f6e2b08ac043c938e3815cdff7a6fd7d87ac269273b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7be22b7a-bd30-4c5c-b938-c1e58f8bfc25\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9be690d6b763dacf73ef1dd42eb495e2367937df20cda0dc3a6bbc8b8f5a0dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34649, \"end_char_idx\": 36651, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\nresults_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>accurate</th>\n      <th>hallucination</th>\n      <th>kb_article</th>\n      <th>chat_history</th>\n      <th>assistant_response</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>true</td>\n      <td>Pass</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>true</td>\n      <td>Pass</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>true</td>\n      <td>Pass</td>\n      <td>PRODUCT FEEDBACK POLICY 1. **Acknowledge Recep...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to let ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>true</td>\n      <td>Pass</td>\n      <td>1. **Acknowledge Reception** - Thank the custo...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>true</td>\n      <td>Pass</td>\n      <td>1. **Acknowledge Reception** - Thank the custo...</td>\n      <td>[{'role': 'user', 'content': 'I wanted to say ...</td>\n      <td>{'role': 'assistant', 'content': 'Thank you fo...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\nresults_df.to_csv('hallucination_results.csv', index=False)\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7be22b7a-bd30-4c5c-b938-c1e58f8bfc25
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb", "title": "Function to set up display options for pandas", "tokens": 9192, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7be22b7a-bd30-4c5c-b938-c1e58f8bfc25\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97a4629d-f346-40a0-81cf-21af6ea5e092\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30a07e775cfdb123d4096dfa09455133a0321794d4d042b198f4ef393c6a1842\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8332cc8f-925c-43a2-8abf-ffa88912630a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Developing_hallucination_guardrails.ipynb\", \"title\": \"Function to set up display options for pandas\", \"tokens\": 9192, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"869b8135da11f0bb744bde9de0612599f82738d6825e11a3e5469c9c528ef311\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 36577, \"end_char_idx\": 39074, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "ref_doc_id": "97a4629d-f346-40a0-81cf-21af6ea5e092", "chroma:document": "```python\nresults_df.to_csv('hallucination_results.csv', index=False)\n\n```\n\n\n```python\ndf = pd.read_csv('hallucination_results.csv')\n\nif 'accurate' not in df.columns or 'hallucination' not in df.columns:\n    print(\"Error: The required columns are not present in the DataFrame.\")\nelse:\n    # Transform values to binary 0/1\n    try:\n        df['accurate'] = df['accurate'].astype(str).str.strip().map(lambda x: 1 if x in ['True', 'true'] else 0)\n        df['hallucination'] = df['hallucination'].str.strip().map(lambda x: 1 if x == 'Pass' else 0)\n        \n    except KeyError as e:\n        print(f\"Mapping error: {e}\")\n\n    # Check for any NaN values after mapping\n    if df['accurate'].isnull().any() or df['hallucination'].isnull().any():\n        print(\"Error: There are NaN values in the mapped columns. Check the input data for unexpected values.\")\n    else:\n        # Calculate precision and recall\n        try:\n            # Precision measures the proportion of correctly identified true positives out of all instances predicted as positive. \n            # Precision = (True Positives) / (True Positives + False Positives)\n            \n            precision = precision_score(df['accurate'], df['hallucination'])\n            \n            # Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.\n            # Recall = (True Positives) / (True Positives + False Negatives)\n            \n            recall = recall_score(df['accurate'], df['hallucination'])\n            \n            \n            print(f\"\\nPrecision: {precision:.2f} (Precision measures the proportion of correctly identified true positives out of all instances predicted as positive.), \"\n                  f\"\\nRecall: {recall:.2f} (Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.)\")\n\n        except ValueError as e:\n            print(f\"Error in calculating precision and recall: {e}\")\n```\n\n    \n    Precision: 0.97 (Precision measures the proportion of correctly identified true positives out of all instances predicted as positive.), \n    Recall: 1.00 (Recall measures the proportion of correctly identified true positives out of all actual positive instances in the dataset.)\n\n\nFrom the results above we can see the program is performing well with a high precision and recall metric. This means that the guardrails are able to accurately identify hallucinations in the model outputs."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 913de36b-d722-44ce-b1c0-b110e6609aa8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"913de36b-d722-44ce-b1c0-b110e6609aa8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c57a679-2f0e-4b78-9d24-0952df264da1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"798d28dd6ec1e1318a81f87696398bf8e020b2a1926af9c6fe164db344ecc822\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2988, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "# Customizing embeddings\n\nThis notebook demonstrates one way to customize OpenAI embeddings to a particular task.\n\nThe input is training data in the form of [text_1, text_2, label] where label is +1 if the pairs are similar and -1 if the pairs are dissimilar.\n\nThe output is a matrix that you can use to multiply your embeddings. The product of this multiplication is a 'custom embedding' that will better emphasize aspects of the text relevant to your use case. In binary classification use cases, we've seen error rates drop by as much as 50%.\n\nIn the following example, I use 1,000 sentence pairs picked from the SNLI corpus. Each pair of sentences are logically entailed (i.e., one implies the other). These pairs are our positives (label = 1). We generate synthetic negatives by combining sentences from different pairs, which are presumed to not be logically entailed (label = -1).\n\nFor a clustering use case, you can generate positives by creating pairs from texts in the same clusters and generate negatives by creating pairs from sentences in different clusters.\n\nWith other data sets, we have seen decent improvement with as little as ~100 training examples. Of course, performance will be better with  more examples.\n\n# 0. Imports\n\n\n```python\n# imports\nfrom typing import List, Tuple  # for type hints\n\nimport numpy as np  # for manipulating arrays\nimport pandas as pd  # for manipulating data in dataframes\nimport pickle  # for saving the embeddings cache\nimport plotly.express as px  # for plots\nimport random  # for generating run IDs\nfrom sklearn.model_selection import train_test_split  # for splitting train & test data\nimport torch  # for matrix optimization\n\nfrom utils.embeddings_utils import get_embedding, cosine_similarity  # for embeddings\n\n```\n\n## 1. Inputs\n\nMost inputs are here. The key things to change are where to load your datset from, where to save a cache of embeddings to, and which embedding engine you want to use.\n\nDepending on how your data is formatted, you'll want to rewrite the process_input_data function.\n\n\n```python\n# input parameters\nembedding_cache_path = \"data/snli_embedding_cache.pkl\"  # embeddings will be saved/loaded here\ndefault_embedding_engine = \"text-embedding-3-small\"\nnum_pairs_to_embed = 1000  # 1000 is arbitrary\nlocal_dataset_path = \"data/snli_1.0_train_2k.csv\"  # download from: https://nlp.stanford.edu/projects/snli/\n\n\ndef process_input_data(df: pd.DataFrame) -> pd.DataFrame:\n    # you can customize this to preprocess your own dataset\n    # output should be a dataframe with 3 columns: text_1, text_2, label (1 for similar, -1 for dissimilar)\n    df[\"label\"] = df[\"gold_label\"]\n    df = df[df[\"label\"].isin([\"entailment\"])]\n    df[\"label\"] = df[\"label\"].apply(lambda x: {\"entailment\": 1, \"contradiction\": -1}[x])\n    df = df.rename(columns={\"sentence1\": \"text_1\", \"sentence2\": \"text_2\"})\n    df = df[[\"text_1\", \"text_2\", \"label\"]]\n    df = df.head(num_pairs_to_embed)\n    return df\n\n```\n\n## 2. Load and process input data"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1c57a679-2f0e-4b78-9d24-0952df264da1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1c57a679-2f0e-4b78-9d24-0952df264da1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"913de36b-d722-44ce-b1c0-b110e6609aa8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"54a2c2cd78734027c33fa9c5239bcdb1ad45e1cb8d7e38292e5b562a23e2a515\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"744b9e01-6dcc-4e0a-836f-e8997d96fd0b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"61efe661b3f18cafd128455f96308ba73a221d00e569e918893fd90d7a125caf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2051, \"end_char_idx\": 3671, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "```python\n# input parameters\nembedding_cache_path = \"data/snli_embedding_cache.pkl\"  # embeddings will be saved/loaded here\ndefault_embedding_engine = \"text-embedding-3-small\"\nnum_pairs_to_embed = 1000  # 1000 is arbitrary\nlocal_dataset_path = \"data/snli_1.0_train_2k.csv\"  # download from: https://nlp.stanford.edu/projects/snli/\n\n\ndef process_input_data(df: pd.DataFrame) -> pd.DataFrame:\n    # you can customize this to preprocess your own dataset\n    # output should be a dataframe with 3 columns: text_1, text_2, label (1 for similar, -1 for dissimilar)\n    df[\"label\"] = df[\"gold_label\"]\n    df = df[df[\"label\"].isin([\"entailment\"])]\n    df[\"label\"] = df[\"label\"].apply(lambda x: {\"entailment\": 1, \"contradiction\": -1}[x])\n    df = df.rename(columns={\"sentence1\": \"text_1\", \"sentence2\": \"text_2\"})\n    df = df[[\"text_1\", \"text_2\", \"label\"]]\n    df = df.head(num_pairs_to_embed)\n    return df\n\n```\n\n## 2. Load and process input data\n\n\n```python\n# load data\ndf = pd.read_csv(local_dataset_path)\n\n# process input data\ndf = process_input_data(df)  # this demonstrates training data containing only positives\n\n# view data\ndf.head()\n\n```\n\n    /var/folders/r4/x3kdvs816995fnnph2gdpwp40000gn/T/ipykernel_17509/1977422881.py:13: SettingWithCopyWarning: \n    A value is trying to be set on a copy of a slice from a DataFrame.\n    Try using .loc[row_indexer,col_indexer] = value instead\n    \n    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n      df[\"label\"] = df[\"label\"].apply(lambda x: {\"entailment\": 1, \"contradiction\": -1}[x])"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 744b9e01-6dcc-4e0a-836f-e8997d96fd0b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"744b9e01-6dcc-4e0a-836f-e8997d96fd0b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c57a679-2f0e-4b78-9d24-0952df264da1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c60272a4947f71f163371c04bc6c9c368b5cad0fdddc73e31db390aab63c8860\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cafee84b-90fc-46f4-81dc-d4ac25139826\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"de2b487346747393ec31512bb5ae7fe22c336b63e0ea6b96bbf62ef9fed28345\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2384, \"end_char_idx\": 4943, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "def process_input_data(df: pd.DataFrame) -> pd.DataFrame:\n    # you can customize this to preprocess your own dataset\n    # output should be a dataframe with 3 columns: text_1, text_2, label (1 for similar, -1 for dissimilar)\n    df[\"label\"] = df[\"gold_label\"]\n    df = df[df[\"label\"].isin([\"entailment\"])]\n    df[\"label\"] = df[\"label\"].apply(lambda x: {\"entailment\": 1, \"contradiction\": -1}[x])\n    df = df.rename(columns={\"sentence1\": \"text_1\", \"sentence2\": \"text_2\"})\n    df = df[[\"text_1\", \"text_2\", \"label\"]]\n    df = df.head(num_pairs_to_embed)\n    return df\n\n```\n\n## 2. Load and process input data\n\n\n```python\n# load data\ndf = pd.read_csv(local_dataset_path)\n\n# process input data\ndf = process_input_data(df)  # this demonstrates training data containing only positives\n\n# view data\ndf.head()\n\n```\n\n    /var/folders/r4/x3kdvs816995fnnph2gdpwp40000gn/T/ipykernel_17509/1977422881.py:13: SettingWithCopyWarning: \n    A value is trying to be set on a copy of a slice from a DataFrame.\n    Try using .loc[row_indexer,col_indexer] = value instead\n    \n    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n      df[\"label\"] = df[\"label\"].apply(lambda x: {\"entailment\": 1, \"contradiction\": -1}[x])\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>text_1</th>\n      <th>text_2</th>\n      <th>label</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2</th>\n      <td>A person on a horse jumps over a broken down a...</td>\n      <td>A person is outdoors, on a horse.</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Children smiling and waving at camera</td>\n      <td>There are children present</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>A boy is jumping on skateboard in the middle o...</td>\n      <td>The boy does a skateboarding trick.</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>Two blond women are hugging one another.</td>\n      <td>There are women showing affection.</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>A few people in a restaurant setting, one of t...</td>\n      <td>The diners are at a restaurant.</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cafee84b-90fc-46f4-81dc-d4ac25139826
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cafee84b-90fc-46f4-81dc-d4ac25139826\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"744b9e01-6dcc-4e0a-836f-e8997d96fd0b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7a937fba3872b8f1361d2d57ec5890a51f92390be1e6186f99ec9716bb9983c6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8af12ab1-e151-4baf-b549-09339da21059\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"712d25e4a4a44479f4d192dac9ed45c8e5c821be93bcff44ca6debef160ff2dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4947, \"end_char_idx\": 8002, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "## 3. Split data into training test sets\n\nNote that it's important to split data into training and test sets *before* generating synethetic negatives or positives. You don't want any text strings in the training data to show up in the test data. If there's contamination, the test metrics will look better than they'll actually be in production.\n\n\n```python\n# split data into train and test sets\ntest_fraction = 0.5  # 0.5 is fairly arbitrary\nrandom_seed = 123  # random seed is arbitrary, but is helpful in reproducibility\ntrain_df, test_df = train_test_split(\n    df, test_size=test_fraction, stratify=df[\"label\"], random_state=random_seed\n)\ntrain_df.loc[:, \"dataset\"] = \"train\"\ntest_df.loc[:, \"dataset\"] = \"test\"\n\n```\n\n## 4. Generate synthetic negatives\n\nThis is another piece of the code that you will need to modify to match your use case.\n\nIf you have data with positives and negatives, you can skip this section.\n\nIf you have data with only positives, you can mostly keep it as is, where it generates negatives only.\n\nIf you have multiclass data, you will want to generate both positives and negatives. The positives can be pairs of text that share labels, and the negatives can be pairs of text that do not share labels.\n\nThe final output should be a dataframe with text pairs, where each pair is labeled -1 or 1.\n\n\n```python\n# generate negatives\ndef dataframe_of_negatives(dataframe_of_positives: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"Return dataframe of negative pairs made by combining elements of positive pairs.\"\"\"\n    texts = set(dataframe_of_positives[\"text_1\"].values) | set(\n        dataframe_of_positives[\"text_2\"].values\n    )\n    all_pairs = {(t1, t2) for t1 in texts for t2 in texts if t1 < t2}\n    positive_pairs = set(\n        tuple(text_pair)\n        for text_pair in dataframe_of_positives[[\"text_1\", \"text_2\"]].values\n    )\n    negative_pairs = all_pairs - positive_pairs\n    df_of_negatives = pd.DataFrame(list(negative_pairs), columns=[\"text_1\", \"text_2\"])\n    df_of_negatives[\"label\"] = -1\n    return df_of_negatives\n\n```\n\n\n```python\nnegatives_per_positive = (\n    1  # it will work at higher values too, but more data will be slower\n)\n# generate negatives for training dataset\ntrain_df_negatives = dataframe_of_negatives(train_df)\ntrain_df_negatives[\"dataset\"] = \"train\"\n# generate negatives for test dataset\ntest_df_negatives = dataframe_of_negatives(test_df)\ntest_df_negatives[\"dataset\"] = \"test\"\n# sample negatives and combine with positives\ntrain_df = pd.concat(\n    [\n        train_df,\n        train_df_negatives.sample(\n            n=len(train_df) * negatives_per_positive, random_state=random_seed\n        ),\n    ]\n)\ntest_df = pd.concat(\n    [\n        test_df,\n        test_df_negatives.sample(\n            n=len(test_df) * negatives_per_positive, random_state=random_seed\n        ),\n    ]\n)\n\ndf = pd.concat([train_df, test_df])\n\n```\n\n## 5. Calculate embeddings and cosine similarities\n\nHere, I create a cache to save the embeddings. This is handy so that you don't have to pay again if you want to run the code again."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8af12ab1-e151-4baf-b549-09339da21059
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8af12ab1-e151-4baf-b549-09339da21059\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cafee84b-90fc-46f4-81dc-d4ac25139826\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1061ea08f26604e5455f16c97f2387ed62e9f48508cac24c7b5bba143c32980e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0bd4b40-d86e-402b-ac5b-5922afdf860a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32d10cff9e34cfe24ef59c4e238b8aa40d5f44759d2b6b2234841854f793e8b1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7000, \"end_char_idx\": 10214, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "```python\nnegatives_per_positive = (\n    1  # it will work at higher values too, but more data will be slower\n)\n# generate negatives for training dataset\ntrain_df_negatives = dataframe_of_negatives(train_df)\ntrain_df_negatives[\"dataset\"] = \"train\"\n# generate negatives for test dataset\ntest_df_negatives = dataframe_of_negatives(test_df)\ntest_df_negatives[\"dataset\"] = \"test\"\n# sample negatives and combine with positives\ntrain_df = pd.concat(\n    [\n        train_df,\n        train_df_negatives.sample(\n            n=len(train_df) * negatives_per_positive, random_state=random_seed\n        ),\n    ]\n)\ntest_df = pd.concat(\n    [\n        test_df,\n        test_df_negatives.sample(\n            n=len(test_df) * negatives_per_positive, random_state=random_seed\n        ),\n    ]\n)\n\ndf = pd.concat([train_df, test_df])\n\n```\n\n## 5. Calculate embeddings and cosine similarities\n\nHere, I create a cache to save the embeddings. This is handy so that you don't have to pay again if you want to run the code again.\n\n\n```python\n# establish a cache of embeddings to avoid recomputing\n# cache is a dict of tuples (text, engine) -> embedding\ntry:\n    with open(embedding_cache_path, \"rb\") as f:\n        embedding_cache = pickle.load(f)\nexcept FileNotFoundError:\n    precomputed_embedding_cache_path = \"https://cdn.openai.com/API/examples/data/snli_embedding_cache.pkl\"\n    embedding_cache = pd.read_pickle(precomputed_embedding_cache_path)\n\n\n# this function will get embeddings from the cache and save them there afterward\ndef get_embedding_with_cache(\n    text: str,\n    engine: str = default_embedding_engine,\n    embedding_cache: dict = embedding_cache,\n    embedding_cache_path: str = embedding_cache_path,\n) -> list:\n    if (text, engine) not in embedding_cache.keys():\n        # if not in cache, call API to get embedding\n        embedding_cache[(text, engine)] = get_embedding(text, engine)\n        # save embeddings cache to disk after each update\n        with open(embedding_cache_path, \"wb\") as embedding_cache_file:\n            pickle.dump(embedding_cache, embedding_cache_file)\n    return embedding_cache[(text, engine)]\n\n\n# create column of embeddings\nfor column in [\"text_1\", \"text_2\"]:\n    df[f\"{column}_embedding\"] = df[column].apply(get_embedding_with_cache)\n\n# create column of cosine similarity between embeddings\ndf[\"cosine_similarity\"] = df.apply(\n    lambda row: cosine_similarity(row[\"text_1_embedding\"], row[\"text_2_embedding\"]),\n    axis=1,\n)\n\n```\n\n## 6. Plot distribution of cosine similarity\n\nHere we measure similarity of text using cosine similarity. In our experience, most distance functions (L1, L2, cosine similarity) all work about the same. Note that our embeddings are already normalized to length 1, so cosine similarity is equivalent to dot product.\n\nThe graphs show how much the overlap there is between the distribution of cosine similarities for similar and dissimilar pairs. If there is a high amount of overlap, that means there are some dissimilar pairs with greater cosine similarity than some similar pairs.\n\nThe accuracy I compute is the accuracy of a simple rule that predicts 'similar (1)' if the cosine similarity is above some threshold X and otherwise predicts 'dissimilar (0)'."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e0bd4b40-d86e-402b-ac5b-5922afdf860a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e0bd4b40-d86e-402b-ac5b-5922afdf860a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8af12ab1-e151-4baf-b549-09339da21059\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5a95ffbaab6bea56c4527bad834cf2bfea5604acbb0cdb02dee731bd639669e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee6faec0-a734-4777-99d7-d3a8bac71977\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2d1c48c3c078f08a4c903c1b6f58c8f32a8461e688509c6467e3dc7fb2dbae0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8426, \"end_char_idx\": 11646, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "# this function will get embeddings from the cache and save them there afterward\ndef get_embedding_with_cache(\n    text: str,\n    engine: str = default_embedding_engine,\n    embedding_cache: dict = embedding_cache,\n    embedding_cache_path: str = embedding_cache_path,\n) -> list:\n    if (text, engine) not in embedding_cache.keys():\n        # if not in cache, call API to get embedding\n        embedding_cache[(text, engine)] = get_embedding(text, engine)\n        # save embeddings cache to disk after each update\n        with open(embedding_cache_path, \"wb\") as embedding_cache_file:\n            pickle.dump(embedding_cache, embedding_cache_file)\n    return embedding_cache[(text, engine)]\n\n\n# create column of embeddings\nfor column in [\"text_1\", \"text_2\"]:\n    df[f\"{column}_embedding\"] = df[column].apply(get_embedding_with_cache)\n\n# create column of cosine similarity between embeddings\ndf[\"cosine_similarity\"] = df.apply(\n    lambda row: cosine_similarity(row[\"text_1_embedding\"], row[\"text_2_embedding\"]),\n    axis=1,\n)\n\n```\n\n## 6. Plot distribution of cosine similarity\n\nHere we measure similarity of text using cosine similarity. In our experience, most distance functions (L1, L2, cosine similarity) all work about the same. Note that our embeddings are already normalized to length 1, so cosine similarity is equivalent to dot product.\n\nThe graphs show how much the overlap there is between the distribution of cosine similarities for similar and dissimilar pairs. If there is a high amount of overlap, that means there are some dissimilar pairs with greater cosine similarity than some similar pairs.\n\nThe accuracy I compute is the accuracy of a simple rule that predicts 'similar (1)' if the cosine similarity is above some threshold X and otherwise predicts 'dissimilar (0)'.\n\n\n```python\n# calculate accuracy (and its standard error) of predicting label=1 if similarity>x\n# x is optimized by sweeping from -1 to 1 in steps of 0.01\ndef accuracy_and_se(cosine_similarity: float, labeled_similarity: int) -> Tuple[float]:\n    accuracies = []\n    for threshold_thousandths in range(-1000, 1000, 1):\n        threshold = threshold_thousandths / 1000\n        total = 0\n        correct = 0\n        for cs, ls in zip(cosine_similarity, labeled_similarity):\n            total += 1\n            if cs > threshold:\n                prediction = 1\n            else:\n                prediction = -1\n            if prediction == ls:\n                correct += 1\n        accuracy = correct / total\n        accuracies.append(accuracy)\n    a = max(accuracies)\n    n = len(cosine_similarity)\n    standard_error = (a * (1 - a) / n) ** 0.5  # standard error of binomial\n    return a, standard_error\n\n\n# check that training and test sets are balanced\npx.histogram(\n    df,\n    x=\"cosine_similarity\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\nfor dataset in [\"train\", \"test\"]:\n    data = df[df[\"dataset\"] == dataset]\n    a, se = accuracy_and_se(data[\"cosine_similarity\"], data[\"label\"])\n    print(f\"{dataset} accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n```\n\n\n\n    train accuracy: 89.1% \u00b1 2.4%\n    test accuracy: 88.8% \u00b1 2.4%\n\n\n## 7. Optimize the matrix using the training data provided"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ee6faec0-a734-4777-99d7-d3a8bac71977
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ee6faec0-a734-4777-99d7-d3a8bac71977\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0bd4b40-d86e-402b-ac5b-5922afdf860a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ded3f2271a88d2f58fb170d986db2abbcbfb5be5ccd78f39f4059a1c55700149\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c59e858-0a47-445f-b1db-647d964c0897\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ad13287483ae83e9fa045d0447069ef883826b1d2ed4a0f48bdf5081ac695b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10217, \"end_char_idx\": 13261, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "```python\n# calculate accuracy (and its standard error) of predicting label=1 if similarity>x\n# x is optimized by sweeping from -1 to 1 in steps of 0.01\ndef accuracy_and_se(cosine_similarity: float, labeled_similarity: int) -> Tuple[float]:\n    accuracies = []\n    for threshold_thousandths in range(-1000, 1000, 1):\n        threshold = threshold_thousandths / 1000\n        total = 0\n        correct = 0\n        for cs, ls in zip(cosine_similarity, labeled_similarity):\n            total += 1\n            if cs > threshold:\n                prediction = 1\n            else:\n                prediction = -1\n            if prediction == ls:\n                correct += 1\n        accuracy = correct / total\n        accuracies.append(accuracy)\n    a = max(accuracies)\n    n = len(cosine_similarity)\n    standard_error = (a * (1 - a) / n) ** 0.5  # standard error of binomial\n    return a, standard_error\n\n\n# check that training and test sets are balanced\npx.histogram(\n    df,\n    x=\"cosine_similarity\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\nfor dataset in [\"train\", \"test\"]:\n    data = df[df[\"dataset\"] == dataset]\n    a, se = accuracy_and_se(data[\"cosine_similarity\"], data[\"label\"])\n    print(f\"{dataset} accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n```\n\n\n\n    train accuracy: 89.1% \u00b1 2.4%\n    test accuracy: 88.8% \u00b1 2.4%\n\n\n## 7. Optimize the matrix using the training data provided\n\n\n```python\ndef embedding_multiplied_by_matrix(\n    embedding: List[float], matrix: torch.tensor\n) -> np.array:\n    embedding_tensor = torch.tensor(embedding).float()\n    modified_embedding = embedding_tensor @ matrix\n    modified_embedding = modified_embedding.detach().numpy()\n    return modified_embedding\n\n\n# compute custom embeddings and new cosine similarities\ndef apply_matrix_to_embeddings_dataframe(matrix: torch.tensor, df: pd.DataFrame):\n    for column in [\"text_1_embedding\", \"text_2_embedding\"]:\n        df[f\"{column}_custom\"] = df[column].apply(\n            lambda x: embedding_multiplied_by_matrix(x, matrix)\n        )\n    df[\"cosine_similarity_custom\"] = df.apply(\n        lambda row: cosine_similarity(\n            row[\"text_1_embedding_custom\"], row[\"text_2_embedding_custom\"]\n        ),\n        axis=1,\n    )\n\n```\n\n\n```python\ndef optimize_matrix(\n    modified_embedding_length: int = 2048,  # in my brief experimentation, bigger was better (2048 is length of babbage encoding)\n    batch_size: int = 100,\n    max_epochs: int = 100,\n    learning_rate: float = 100.0,  # seemed to work best when similar to batch size - feel free to try a range of values\n    dropout_fraction: float = 0.0,  # in my testing, dropout helped by a couple percentage points (definitely not necessary)\n    df: pd.DataFrame = df,\n    print_progress: bool = True,\n    save_results: bool = True,\n) -> torch.tensor:\n    \"\"\"Return matrix optimized to minimize loss on training data.\"\"\"\n    run_id = random.randint(0, 2 ** 31 - 1)  # (range is arbitrary)\n    # convert from dataframe to torch tensors\n    # e is for embedding,"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8c59e858-0a47-445f-b1db-647d964c0897
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8c59e858-0a47-445f-b1db-647d964c0897\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee6faec0-a734-4777-99d7-d3a8bac71977\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"205590afc262242470c8243ab6d888f8bcfe205ca968e9c635e823f0650e6b1f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dc44cda4-a1bf-49d1-b3d2-c0360eb1afb5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"832f50598869350d9fed1fc8b8af62bcab0ea233d6c829281dba770bdb6aca6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11588, \"end_char_idx\": 14616, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "## 7. Optimize the matrix using the training data provided\n\n\n```python\ndef embedding_multiplied_by_matrix(\n    embedding: List[float], matrix: torch.tensor\n) -> np.array:\n    embedding_tensor = torch.tensor(embedding).float()\n    modified_embedding = embedding_tensor @ matrix\n    modified_embedding = modified_embedding.detach().numpy()\n    return modified_embedding\n\n\n# compute custom embeddings and new cosine similarities\ndef apply_matrix_to_embeddings_dataframe(matrix: torch.tensor, df: pd.DataFrame):\n    for column in [\"text_1_embedding\", \"text_2_embedding\"]:\n        df[f\"{column}_custom\"] = df[column].apply(\n            lambda x: embedding_multiplied_by_matrix(x, matrix)\n        )\n    df[\"cosine_similarity_custom\"] = df.apply(\n        lambda row: cosine_similarity(\n            row[\"text_1_embedding_custom\"], row[\"text_2_embedding_custom\"]\n        ),\n        axis=1,\n    )\n\n```\n\n\n```python\ndef optimize_matrix(\n    modified_embedding_length: int = 2048,  # in my brief experimentation, bigger was better (2048 is length of babbage encoding)\n    batch_size: int = 100,\n    max_epochs: int = 100,\n    learning_rate: float = 100.0,  # seemed to work best when similar to batch size - feel free to try a range of values\n    dropout_fraction: float = 0.0,  # in my testing, dropout helped by a couple percentage points (definitely not necessary)\n    df: pd.DataFrame = df,\n    print_progress: bool = True,\n    save_results: bool = True,\n) -> torch.tensor:\n    \"\"\"Return matrix optimized to minimize loss on training data.\"\"\"\n    run_id = random.randint(0, 2 ** 31 - 1)  # (range is arbitrary)\n    # convert from dataframe to torch tensors\n    # e is for embedding, s for similarity label\n    def tensors_from_dataframe(\n        df: pd.DataFrame,\n        embedding_column_1: str,\n        embedding_column_2: str,\n        similarity_label_column: str,\n    ) -> Tuple[torch.tensor]:\n        e1 = np.stack(np.array(df[embedding_column_1].values))\n        e2 = np.stack(np.array(df[embedding_column_2].values))\n        s = np.stack(np.array(df[similarity_label_column].astype(\"float\").values))\n\n        e1 = torch.from_numpy(e1).float()\n        e2 = torch.from_numpy(e2).float()\n        s = torch.from_numpy(s).float()\n\n        return e1, e2, s\n\n    e1_train, e2_train, s_train = tensors_from_dataframe(\n        df[df[\"dataset\"] == \"train\"], \"text_1_embedding\", \"text_2_embedding\", \"label\"\n    )\n    e1_test, e2_test, s_test = tensors_from_dataframe(\n        df[df[\"dataset\"] == \"test\"], \"text_1_embedding\", \"text_2_embedding\", \"label\"\n    )\n\n    # create dataset and loader\n    dataset = torch.utils.data.TensorDataset(e1_train, e2_train, s_train)\n    train_loader = torch.utils.data.DataLoader(\n        dataset, batch_size=batch_size, shuffle=True\n    )\n\n    # define model (similarity of projected embeddings)\n    def model(embedding_1, embedding_2, matrix, dropout_fraction=dropout_fraction):\n        e1 = torch.nn.functional.dropout(embedding_1, p=dropout_fraction)\n        e2 = torch.nn.functional.dropout(embedding_2,"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dc44cda4-a1bf-49d1-b3d2-c0360eb1afb5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dc44cda4-a1bf-49d1-b3d2-c0360eb1afb5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8c59e858-0a47-445f-b1db-647d964c0897\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e12968aba896d6533772f7936f1b5bae9f57c226e64d20b44fb8f3434aafd9b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a7f8a91c-6421-4be1-ab9f-e53c231b7b4e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0a8bef462b8ea870f53cc28b92714b917ef002ef1baaee54e393bf4abacf9e39\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13262, \"end_char_idx\": 16563, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "s for similarity label\n    def tensors_from_dataframe(\n        df: pd.DataFrame,\n        embedding_column_1: str,\n        embedding_column_2: str,\n        similarity_label_column: str,\n    ) -> Tuple[torch.tensor]:\n        e1 = np.stack(np.array(df[embedding_column_1].values))\n        e2 = np.stack(np.array(df[embedding_column_2].values))\n        s = np.stack(np.array(df[similarity_label_column].astype(\"float\").values))\n\n        e1 = torch.from_numpy(e1).float()\n        e2 = torch.from_numpy(e2).float()\n        s = torch.from_numpy(s).float()\n\n        return e1, e2, s\n\n    e1_train, e2_train, s_train = tensors_from_dataframe(\n        df[df[\"dataset\"] == \"train\"], \"text_1_embedding\", \"text_2_embedding\", \"label\"\n    )\n    e1_test, e2_test, s_test = tensors_from_dataframe(\n        df[df[\"dataset\"] == \"test\"], \"text_1_embedding\", \"text_2_embedding\", \"label\"\n    )\n\n    # create dataset and loader\n    dataset = torch.utils.data.TensorDataset(e1_train, e2_train, s_train)\n    train_loader = torch.utils.data.DataLoader(\n        dataset, batch_size=batch_size, shuffle=True\n    )\n\n    # define model (similarity of projected embeddings)\n    def model(embedding_1, embedding_2, matrix, dropout_fraction=dropout_fraction):\n        e1 = torch.nn.functional.dropout(embedding_1, p=dropout_fraction)\n        e2 = torch.nn.functional.dropout(embedding_2, p=dropout_fraction)\n        modified_embedding_1 = e1 @ matrix  # @ is matrix multiplication\n        modified_embedding_2 = e2 @ matrix\n        similarity = torch.nn.functional.cosine_similarity(\n            modified_embedding_1, modified_embedding_2\n        )\n        return similarity\n\n    # define loss function to minimize\n    def mse_loss(predictions, targets):\n        difference = predictions - targets\n        return torch.sum(difference * difference) / difference.numel()\n\n    # initialize projection matrix\n    embedding_length = len(df[\"text_1_embedding\"].values[0])\n    matrix = torch.randn(\n        embedding_length, modified_embedding_length, requires_grad=True\n    )\n\n    epochs, types, losses, accuracies, matrices = [], [], [], [], []\n    for epoch in range(1, 1 + max_epochs):\n        # iterate through training dataloader\n        for a, b, actual_similarity in train_loader:\n            # generate prediction\n            predicted_similarity = model(a, b, matrix)\n            # get loss and perform backpropagation\n            loss = mse_loss(predicted_similarity, actual_similarity)\n            loss.backward()\n            # update the weights\n            with torch.no_grad():\n                matrix -= matrix.grad * learning_rate\n                # set gradients to zero\n                matrix.grad.zero_()\n        # calculate test loss\n        test_predictions = model(e1_test, e2_test, matrix)\n        test_loss = mse_loss(test_predictions, s_test)\n\n        # compute custom embeddings and new cosine similarities\n        apply_matrix_to_embeddings_dataframe(matrix, df)\n\n        # calculate test accuracy\n        for dataset in [\"train\", \"test\"]:\n            data = df[df[\"dataset\"] == dataset]\n            a, se = accuracy_and_se(data[\"cosine_similarity_custom\"], data[\"label\"])\n\n            # record results of each epoch\n            epochs.append(epoch)\n            types.append(dataset)\n            losses.append(loss."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a7f8a91c-6421-4be1-ab9f-e53c231b7b4e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a7f8a91c-6421-4be1-ab9f-e53c231b7b4e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dc44cda4-a1bf-49d1-b3d2-c0360eb1afb5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"189d3a016f33ef76b5f58f2663a76d0c97e1da10ca6a48b0e6f0838979c5e606\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e4c3c76-1ce3-41c9-a684-fd247531c2ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8bc06622347a2f28ae6cf7811f5a8492f371d43392ad4d201fd646fb133b06ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14780, \"end_char_idx\": 18126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "nn.functional.cosine_similarity(\n            modified_embedding_1, modified_embedding_2\n        )\n        return similarity\n\n    # define loss function to minimize\n    def mse_loss(predictions, targets):\n        difference = predictions - targets\n        return torch.sum(difference * difference) / difference.numel()\n\n    # initialize projection matrix\n    embedding_length = len(df[\"text_1_embedding\"].values[0])\n    matrix = torch.randn(\n        embedding_length, modified_embedding_length, requires_grad=True\n    )\n\n    epochs, types, losses, accuracies, matrices = [], [], [], [], []\n    for epoch in range(1, 1 + max_epochs):\n        # iterate through training dataloader\n        for a, b, actual_similarity in train_loader:\n            # generate prediction\n            predicted_similarity = model(a, b, matrix)\n            # get loss and perform backpropagation\n            loss = mse_loss(predicted_similarity, actual_similarity)\n            loss.backward()\n            # update the weights\n            with torch.no_grad():\n                matrix -= matrix.grad * learning_rate\n                # set gradients to zero\n                matrix.grad.zero_()\n        # calculate test loss\n        test_predictions = model(e1_test, e2_test, matrix)\n        test_loss = mse_loss(test_predictions, s_test)\n\n        # compute custom embeddings and new cosine similarities\n        apply_matrix_to_embeddings_dataframe(matrix, df)\n\n        # calculate test accuracy\n        for dataset in [\"train\", \"test\"]:\n            data = df[df[\"dataset\"] == dataset]\n            a, se = accuracy_and_se(data[\"cosine_similarity_custom\"], data[\"label\"])\n\n            # record results of each epoch\n            epochs.append(epoch)\n            types.append(dataset)\n            losses.append(loss.item() if dataset == \"train\" else test_loss.item())\n            accuracies.append(a)\n            matrices.append(matrix.detach().numpy())\n\n            # optionally print accuracies\n            if print_progress is True:\n                print(\n                    f\"Epoch {epoch}/{max_epochs}: {dataset} accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\"\n                )\n\n    data = pd.DataFrame(\n        {\"epoch\": epochs, \"type\": types, \"loss\": losses, \"accuracy\": accuracies}\n    )\n    data[\"run_id\"] = run_id\n    data[\"modified_embedding_length\"] = modified_embedding_length\n    data[\"batch_size\"] = batch_size\n    data[\"max_epochs\"] = max_epochs\n    data[\"learning_rate\"] = learning_rate\n    data[\"dropout_fraction\"] = dropout_fraction\n    data[\n        \"matrix\"\n    ] = matrices  # saving every single matrix can get big; feel free to delete/change\n    if save_results is True:\n        data.to_csv(f\"{run_id}_optimization_results.csv\", index=False)\n\n    return data\n\n```\n\n\n```python\n# example hyperparameter search\n# I recommend starting with max_epochs=10 while initially exploring\nresults = []\nmax_epochs = 30\ndropout_fraction = 0.2\nfor batch_size, learning_rate in [(10, 10), (100, 100), (1000, 1000)]:\n    result = optimize_matrix(\n        batch_size=batch_size,\n        learning_rate=learning_rate,\n        max_epochs=max_epochs,\n        dropout_fraction=dropout_fraction,\n        save_results=False,\n    )\n    results.append(result)\n\n```\n\n    Epoch 1/30: train accuracy: 89.1% \u00b1 2.4%\n    Epoch 1/30: test accuracy: 88.4% \u00b1 2.4%\n    Epoch 2/30: train accuracy: 89."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7e4c3c76-1ce3-41c9-a684-fd247531c2ac
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7e4c3c76-1ce3-41c9-a684-fd247531c2ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a7f8a91c-6421-4be1-ab9f-e53c231b7b4e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a4cd34ff5681f1f6b15bac5cc0598a3f61f3ea98221a3f90070d7b5f0c667309\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78431243-fc41-4403-9ace-2a1ae1822630\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7dab543ded95c0e726236dbcb003a36e0f5e5de17756fc0e02c53744090e70f2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16607, \"end_char_idx\": 18891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "item())\n            accuracies.append(a)\n            matrices.append(matrix.detach().numpy())\n\n            # optionally print accuracies\n            if print_progress is True:\n                print(\n                    f\"Epoch {epoch}/{max_epochs}: {dataset} accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\"\n                )\n\n    data = pd.DataFrame(\n        {\"epoch\": epochs, \"type\": types, \"loss\": losses, \"accuracy\": accuracies}\n    )\n    data[\"run_id\"] = run_id\n    data[\"modified_embedding_length\"] = modified_embedding_length\n    data[\"batch_size\"] = batch_size\n    data[\"max_epochs\"] = max_epochs\n    data[\"learning_rate\"] = learning_rate\n    data[\"dropout_fraction\"] = dropout_fraction\n    data[\n        \"matrix\"\n    ] = matrices  # saving every single matrix can get big; feel free to delete/change\n    if save_results is True:\n        data.to_csv(f\"{run_id}_optimization_results.csv\", index=False)\n\n    return data\n\n```\n\n\n```python\n# example hyperparameter search\n# I recommend starting with max_epochs=10 while initially exploring\nresults = []\nmax_epochs = 30\ndropout_fraction = 0.2\nfor batch_size, learning_rate in [(10, 10), (100, 100), (1000, 1000)]:\n    result = optimize_matrix(\n        batch_size=batch_size,\n        learning_rate=learning_rate,\n        max_epochs=max_epochs,\n        dropout_fraction=dropout_fraction,\n        save_results=False,\n    )\n    results.append(result)\n\n```\n\n    Epoch 1/30: train accuracy: 89.1% \u00b1 2.4%\n    Epoch 1/30: test accuracy: 88.4% \u00b1 2.4%\n    Epoch 2/30: train accuracy: 89.5% \u00b1 2.3%\n    Epoch 2/30: test accuracy: 88.8% \u00b1 2.4%\n    Epoch 3/30: train accuracy: 90.6% \u00b1 2.2%\n    Epoch 3/30: test accuracy: 89.3% \u00b1 2.3%\n    Epoch 4/30: train accuracy: 91.2% \u00b1 2.2%\n    Epoch 4/30: test accuracy: 89.7% \u00b1 2.3%\n    Epoch 5/30: train accuracy: 91.5% \u00b1 2.1%\n    Epoch 5/30: test accuracy: 90.0% \u00b1 2.3%\n    Epoch 6/30: train accuracy: 91.9% \u00b1 2.1%\n    Epoch 6/30: test accuracy: 90.4% \u00b1 2.2%\n    Epoch 7/30: train accuracy: 92.2% \u00b1 2.0%\n    Epoch 7/30: test accuracy: 90.7% \u00b1 2.2%\n    Epoch 8/30: train accuracy: 92.7% \u00b1 2.0%\n    Epoch 8/30: test accuracy: 90.9% \u00b1 2.2%\n    Epoch 9/30: train accuracy: 92.7% \u00b1 2.0%\n    Epoch 9/30: test accuracy: 91.0% \u00b1 2.2%\n    Epoch 10/30: train accuracy: 93.0% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 91.6% \u00b1 2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 78431243-fc41-4403-9ace-2a1ae1822630
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"78431243-fc41-4403-9ace-2a1ae1822630\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e4c3c76-1ce3-41c9-a684-fd247531c2ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"bc51577765872c1ddd007a27835ef6d66d3175ea56d0837675b2e7e3d3994dcb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b04cd45d-d77a-419b-9c27-37dcf7a85125\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd3302f74fb7d1d159862e481630eeee78f2c5ec454cff3f4bb78998baf6a503\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18044, \"end_char_idx\": 19710, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "4%\n    Epoch 1/30: test accuracy: 88.4% \u00b1 2.4%\n    Epoch 2/30: train accuracy: 89.5% \u00b1 2.3%\n    Epoch 2/30: test accuracy: 88.8% \u00b1 2.4%\n    Epoch 3/30: train accuracy: 90.6% \u00b1 2.2%\n    Epoch 3/30: test accuracy: 89.3% \u00b1 2.3%\n    Epoch 4/30: train accuracy: 91.2% \u00b1 2.2%\n    Epoch 4/30: test accuracy: 89.7% \u00b1 2.3%\n    Epoch 5/30: train accuracy: 91.5% \u00b1 2.1%\n    Epoch 5/30: test accuracy: 90.0% \u00b1 2.3%\n    Epoch 6/30: train accuracy: 91.9% \u00b1 2.1%\n    Epoch 6/30: test accuracy: 90.4% \u00b1 2.2%\n    Epoch 7/30: train accuracy: 92.2% \u00b1 2.0%\n    Epoch 7/30: test accuracy: 90.7% \u00b1 2.2%\n    Epoch 8/30: train accuracy: 92.7% \u00b1 2.0%\n    Epoch 8/30: test accuracy: 90.9% \u00b1 2.2%\n    Epoch 9/30: train accuracy: 92.7% \u00b1 2.0%\n    Epoch 9/30: test accuracy: 91.0% \u00b1 2.2%\n    Epoch 10/30: train accuracy: 93.0% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 91.6% \u00b1 2.1%\n    Epoch 11/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 12/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 12/30: test accuracy: 92.1% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 13/30: test accuracy: 92.4% \u00b1 2.0%\n    Epoch 14/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 15/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 16/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 17/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 18/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 18/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 19/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 19/30: test accuracy: 93.1% \u00b1 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b04cd45d-d77a-419b-9c27-37dcf7a85125
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b04cd45d-d77a-419b-9c27-37dcf7a85125\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78431243-fc41-4403-9ace-2a1ae1822630\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"3b0e8e63b94d353a38ada4bf8a0342a48c30b30d64db3154a5545e8ea15987b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c139eec8-bf81-4d07-9404-561903bed49f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e94cd5ad363918e2b3c0bcf82305c0c9a6388adb4b35742adf24c7e71aa78dc2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18846, \"end_char_idx\": 20529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "9%\n    Epoch 10/30: test accuracy: 91.6% \u00b1 2.1%\n    Epoch 11/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 12/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 12/30: test accuracy: 92.1% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 13/30: test accuracy: 92.4% \u00b1 2.0%\n    Epoch 14/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 15/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 16/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 17/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 18/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 18/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 19/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 19/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 20/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 20/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 21/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 22/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.4% \u00b1 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c139eec8-bf81-4d07-9404-561903bed49f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c139eec8-bf81-4d07-9404-561903bed49f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b04cd45d-d77a-419b-9c27-37dcf7a85125\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5fc9d300bf8aea48ca1ecf9c0682ffd9c6c5d15e804d22373371ab1de4073ad9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"376ce533-21c7-44e9-be1c-42d37784547a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b0eca73acf4938d0ce44a9ceec072905f7350a1103e55093a560e63841d67f7b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19665, \"end_char_idx\": 21334, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "8%\n    Epoch 19/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 20/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 20/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 21/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 22/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 29/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 30/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 1/30: train accuracy: 89.7% \u00b1 2.3%\n    Epoch 1/30: test accuracy: 89.1% \u00b1 2.4%\n    Epoch 2/30: train accuracy: 89.8% \u00b1 2.3%\n    Epoch 2/30: test accuracy: 89.9% \u00b1 2.3%\n    Epoch 3/30: train accuracy: 90.3% \u00b1 2.2%\n    Epoch 3/30: test accuracy: 90.0% \u00b1 2.3%\n    Epoch 4/30: train accuracy: 91.0% \u00b1 2.2%\n    Epoch 4/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 5/30: train accuracy: 91.3% \u00b1 2.1%\n    Epoch 5/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 6/30: train accuracy: 91.8% \u00b1 2.1%\n    Epoch 6/30: test accuracy: 90.4% \u00b1 2.2%\n    Epoch 7/30: train accuracy: 92.4% \u00b1 2.0%\n    Epoch 7/30: test accuracy: 91.0% \u00b1 2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 376ce533-21c7-44e9-be1c-42d37784547a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"376ce533-21c7-44e9-be1c-42d37784547a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c139eec8-bf81-4d07-9404-561903bed49f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2c1adcb5f4a8d3a04309a7d68ff581d25bc8de32c3de8d1344b160815cff7bff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a8e4afa-9f2b-4c12-9674-56663187cc79\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"669da3f0e145d80a690a7375cc3143d1f430be5d6baa03bc01d0ff08ce999ac9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20484, \"end_char_idx\": 22149, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "7%\n    Epoch 28/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 29/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 30/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 1/30: train accuracy: 89.7% \u00b1 2.3%\n    Epoch 1/30: test accuracy: 89.1% \u00b1 2.4%\n    Epoch 2/30: train accuracy: 89.8% \u00b1 2.3%\n    Epoch 2/30: test accuracy: 89.9% \u00b1 2.3%\n    Epoch 3/30: train accuracy: 90.3% \u00b1 2.2%\n    Epoch 3/30: test accuracy: 90.0% \u00b1 2.3%\n    Epoch 4/30: train accuracy: 91.0% \u00b1 2.2%\n    Epoch 4/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 5/30: train accuracy: 91.3% \u00b1 2.1%\n    Epoch 5/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 6/30: train accuracy: 91.8% \u00b1 2.1%\n    Epoch 6/30: test accuracy: 90.4% \u00b1 2.2%\n    Epoch 7/30: train accuracy: 92.4% \u00b1 2.0%\n    Epoch 7/30: test accuracy: 91.0% \u00b1 2.2%\n    Epoch 8/30: train accuracy: 92.8% \u00b1 2.0%\n    Epoch 8/30: test accuracy: 91.3% \u00b1 2.1%\n    Epoch 9/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 9/30: test accuracy: 91.6% \u00b1 2.1%\n    Epoch 10/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 91.9% \u00b1 2.1%\n    Epoch 11/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 12/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 12/30: test accuracy: 92.1% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 13/30: test accuracy: 92.4% \u00b1 2.0%\n    Epoch 14/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 92.5% \u00b1 2.0%\n    Epoch 15/30: train accuracy: 93.9% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 16/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 92.8% \u00b1 2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9a8e4afa-9f2b-4c12-9674-56663187cc79
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9a8e4afa-9f2b-4c12-9674-56663187cc79\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"376ce533-21c7-44e9-be1c-42d37784547a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce854f0a2a2869424dc8e282dd186d515a2a4d5648474477542857ab72c1cca7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"116289a4-d08b-42b9-a8b4-e96ed5749446\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b64af1bbb481e55920f1f726d1fd7d935ae3be27495763e0e69b7438cdfd79a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21290, \"end_char_idx\": 22968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "0%\n    Epoch 7/30: test accuracy: 91.0% \u00b1 2.2%\n    Epoch 8/30: train accuracy: 92.8% \u00b1 2.0%\n    Epoch 8/30: test accuracy: 91.3% \u00b1 2.1%\n    Epoch 9/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 9/30: test accuracy: 91.6% \u00b1 2.1%\n    Epoch 10/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 91.9% \u00b1 2.1%\n    Epoch 11/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 12/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 12/30: test accuracy: 92.1% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 13/30: test accuracy: 92.4% \u00b1 2.0%\n    Epoch 14/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 92.5% \u00b1 2.0%\n    Epoch 15/30: train accuracy: 93.9% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 16/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 17/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 18/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 18/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 19/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 19/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 20/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 20/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 21/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 22/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.4% \u00b1 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 116289a4-d08b-42b9-a8b4-e96ed5749446
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"116289a4-d08b-42b9-a8b4-e96ed5749446\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a8e4afa-9f2b-4c12-9674-56663187cc79\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2e439d277ff9892c12ef6f40889c292efde901a19708cbfa607c435cdd206f5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a5283d3e-a463-4b9b-ab64-aab69edb68a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"488abbe0d4090c75449bec40ade5f24e4e5bae1a1aa9513d76903222d1aabff3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22104, \"end_char_idx\": 23779, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "8%\n    Epoch 16/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 17/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 18/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 18/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 19/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 19/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 20/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 20/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 21/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 22/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 29/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 30/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 1/30: train accuracy: 90.7% \u00b1 2.2%\n    Epoch 1/30: test accuracy: 89.9% \u00b1 2.3%\n    Epoch 2/30: train accuracy: 90.9% \u00b1 2.2%\n    Epoch 2/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 3/30: train accuracy: 91.6% \u00b1 2.1%\n    Epoch 3/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 4/30: train accuracy: 92.2% \u00b1 2.0%\n    Epoch 4/30: test accuracy: 90.7% \u00b1 2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a5283d3e-a463-4b9b-ab64-aab69edb68a5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a5283d3e-a463-4b9b-ab64-aab69edb68a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"116289a4-d08b-42b9-a8b4-e96ed5749446\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"bc0bfa368680cfab2a2ac9f5a492f02772d24f845ffa37e3e778af68e51a9243\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ddd8d73e-bae5-424c-b2f2-630892811934\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0054eba0f3e90877392c604cdd9bb99da2f9d1c16b77379b7df145009dde73a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22923, \"end_char_idx\": 24588, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "7%\n    Epoch 25/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 29/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 30/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 1/30: train accuracy: 90.7% \u00b1 2.2%\n    Epoch 1/30: test accuracy: 89.9% \u00b1 2.3%\n    Epoch 2/30: train accuracy: 90.9% \u00b1 2.2%\n    Epoch 2/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 3/30: train accuracy: 91.6% \u00b1 2.1%\n    Epoch 3/30: test accuracy: 90.3% \u00b1 2.2%\n    Epoch 4/30: train accuracy: 92.2% \u00b1 2.0%\n    Epoch 4/30: test accuracy: 90.7% \u00b1 2.2%\n    Epoch 5/30: train accuracy: 92.4% \u00b1 2.0%\n    Epoch 5/30: test accuracy: 91.3% \u00b1 2.1%\n    Epoch 6/30: train accuracy: 92.5% \u00b1 2.0%\n    Epoch 6/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 7/30: train accuracy: 93.0% \u00b1 1.9%\n    Epoch 7/30: test accuracy: 92.2% \u00b1 2.0%\n    Epoch 8/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 8/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 9/30: train accuracy: 93.3% \u00b1 1.9%\n    Epoch 9/30: test accuracy: 92.5% \u00b1 2.0%\n    Epoch 10/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 11/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 12/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 12/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 13/30: test accuracy: 93.0% \u00b1 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ddd8d73e-bae5-424c-b2f2-630892811934
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ddd8d73e-bae5-424c-b2f2-630892811934\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a5283d3e-a463-4b9b-ab64-aab69edb68a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"677aa83d3afbb74c7990b5224a4cc9805a5b797e43e2091db08d5657b7c0ba47\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d1d5b31-0204-4c20-a14b-446bdf5946eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b76d3adb2c023e94972d9bddffa2071993ee481ea38b5c46814e6f89a4a6c4cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23735, \"end_char_idx\": 25407, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "0%\n    Epoch 4/30: test accuracy: 90.7% \u00b1 2.2%\n    Epoch 5/30: train accuracy: 92.4% \u00b1 2.0%\n    Epoch 5/30: test accuracy: 91.3% \u00b1 2.1%\n    Epoch 6/30: train accuracy: 92.5% \u00b1 2.0%\n    Epoch 6/30: test accuracy: 91.8% \u00b1 2.1%\n    Epoch 7/30: train accuracy: 93.0% \u00b1 1.9%\n    Epoch 7/30: test accuracy: 92.2% \u00b1 2.0%\n    Epoch 8/30: train accuracy: 93.1% \u00b1 1.9%\n    Epoch 8/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 9/30: train accuracy: 93.3% \u00b1 1.9%\n    Epoch 9/30: test accuracy: 92.5% \u00b1 2.0%\n    Epoch 10/30: train accuracy: 93.4% \u00b1 1.9%\n    Epoch 10/30: test accuracy: 92.7% \u00b1 2.0%\n    Epoch 11/30: train accuracy: 93.6% \u00b1 1.9%\n    Epoch 11/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 12/30: train accuracy: 93.7% \u00b1 1.8%\n    Epoch 12/30: test accuracy: 92.8% \u00b1 2.0%\n    Epoch 13/30: train accuracy: 94.0% \u00b1 1.8%\n    Epoch 13/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 14/30: train accuracy: 93.9% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 15/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 16/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 17/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 18/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 18/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 19/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 19/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 20/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 20/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 21/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 22/30: test accuracy: 93.4% \u00b1 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7d1d5b31-0204-4c20-a14b-446bdf5946eb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7d1d5b31-0204-4c20-a14b-446bdf5946eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddd8d73e-bae5-424c-b2f2-630892811934\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"65cb0c9eb2de499a781969cb643acfc7562ed0b71620b5c2812cfdac006f64a1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9c13ebf4-3ea9-4d40-a764-afb2b83ed714\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83b5b354a8895a2cde7b302aa1e69a9a9c7437598efeab6fb2a6d2c24fda27a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24543, \"end_char_idx\": 26137, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "8%\n    Epoch 13/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 14/30: train accuracy: 93.9% \u00b1 1.8%\n    Epoch 14/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 15/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 15/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 16/30: train accuracy: 94.2% \u00b1 1.8%\n    Epoch 16/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 17/30: train accuracy: 94.3% \u00b1 1.8%\n    Epoch 17/30: test accuracy: 93.0% \u00b1 1.9%\n    Epoch 18/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 18/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 19/30: train accuracy: 94.5% \u00b1 1.7%\n    Epoch 19/30: test accuracy: 93.1% \u00b1 1.9%\n    Epoch 20/30: train accuracy: 94.6% \u00b1 1.7%\n    Epoch 20/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 21/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 21/30: test accuracy: 93.3% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 22/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 29/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 30/30: test accuracy: 93.6% \u00b1 1.9%"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9c13ebf4-3ea9-4d40-a764-afb2b83ed714
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9c13ebf4-3ea9-4d40-a764-afb2b83ed714\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d1d5b31-0204-4c20-a14b-446bdf5946eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"31ad949db4cf3662d9fc590018b542627b2a670ef8d2752a18e6945087364f7d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0c1690e6-acdc-413c-9fe7-3a266bb1b352\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"331a281f80f8f7f16f667e3db7575f60f3b11de9c94e1ea6e90d47dbba9446f1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25309, \"end_char_idx\": 27162, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "3% \u00b1 1.9%\n    Epoch 22/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 22/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 23/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 23/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 24/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 24/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 25/30: train accuracy: 94.8% \u00b1 1.7%\n    Epoch 25/30: test accuracy: 93.4% \u00b1 1.9%\n    Epoch 26/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 26/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 27/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 27/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 29/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 30/30: test accuracy: 93.6% \u00b1 1.9%\n\n\n\n```python\nruns_df = pd.concat(results)\n\n# plot training loss and test loss over time\npx.line(\n    runs_df,\n    line_group=\"run_id\",\n    x=\"epoch\",\n    y=\"loss\",\n    color=\"type\",\n    hover_data=[\"batch_size\", \"learning_rate\", \"dropout_fraction\"],\n    facet_row=\"learning_rate\",\n    facet_col=\"batch_size\",\n    width=500,\n).show()\n\n# plot accuracy over time\npx.line(\n    runs_df,\n    line_group=\"run_id\",\n    x=\"epoch\",\n    y=\"accuracy\",\n    color=\"type\",\n    hover_data=[\"batch_size\", \"learning_rate\", \"dropout_fraction\"],\n    facet_row=\"learning_rate\",\n    facet_col=\"batch_size\",\n    width=500,\n).show()\n\n```\n\n\n\n\n\n## 8. Plot the before & after, showing the results of the best matrix found during training\n\nThe better the matrix is, the more cleanly it will separate the similar and dissimilar pairs.\n\n\n```python\n# apply result of best run to original data\nbest_run = runs_df.sort_values(by=\"accuracy\", ascending=False).iloc[0]\nbest_matrix = best_run[\"matrix\"]\napply_matrix_to_embeddings_dataframe(best_matrix, df)\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0c1690e6-acdc-413c-9fe7-3a266bb1b352
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0c1690e6-acdc-413c-9fe7-3a266bb1b352\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9c13ebf4-3ea9-4d40-a764-afb2b83ed714\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"70af99ff9b3783cb30e046e5236723a31ea1cb696c1e3f3d68f47761746f917e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e7ffef2-ec54-4744-bc62-c86b09c8ad92\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"129840b964943852d050669f6e8682d9c804487de3d3eb61f04cfbc4ebdbd367\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25855, \"end_char_idx\": 28060, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "6% \u00b1 1.9%\n    Epoch 28/30: train accuracy: 94.9% \u00b1 1.7%\n    Epoch 28/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 29/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 29/30: test accuracy: 93.6% \u00b1 1.9%\n    Epoch 30/30: train accuracy: 95.1% \u00b1 1.6%\n    Epoch 30/30: test accuracy: 93.6% \u00b1 1.9%\n\n\n\n```python\nruns_df = pd.concat(results)\n\n# plot training loss and test loss over time\npx.line(\n    runs_df,\n    line_group=\"run_id\",\n    x=\"epoch\",\n    y=\"loss\",\n    color=\"type\",\n    hover_data=[\"batch_size\", \"learning_rate\", \"dropout_fraction\"],\n    facet_row=\"learning_rate\",\n    facet_col=\"batch_size\",\n    width=500,\n).show()\n\n# plot accuracy over time\npx.line(\n    runs_df,\n    line_group=\"run_id\",\n    x=\"epoch\",\n    y=\"accuracy\",\n    color=\"type\",\n    hover_data=[\"batch_size\", \"learning_rate\", \"dropout_fraction\"],\n    facet_row=\"learning_rate\",\n    facet_col=\"batch_size\",\n    width=500,\n).show()\n\n```\n\n\n\n\n\n## 8. Plot the before & after, showing the results of the best matrix found during training\n\nThe better the matrix is, the more cleanly it will separate the similar and dissimilar pairs.\n\n\n```python\n# apply result of best run to original data\nbest_run = runs_df.sort_values(by=\"accuracy\", ascending=False).iloc[0]\nbest_matrix = best_run[\"matrix\"]\napply_matrix_to_embeddings_dataframe(best_matrix, df)\n\n```\n\n\n```python\n# plot similarity distribution BEFORE customization\npx.histogram(\n    df,\n    x=\"cosine_similarity\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\ntest_df = df[df[\"dataset\"] == \"test\"]\na, se = accuracy_and_se(test_df[\"cosine_similarity\"], test_df[\"label\"])\nprint(f\"Test accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n# plot similarity distribution AFTER customization\npx.histogram(\n    df,\n    x=\"cosine_similarity_custom\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\na, se = accuracy_and_se(test_df[\"cosine_similarity_custom\"], test_df[\"label\"])\nprint(f\"Test accuracy after customization: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n```\n\n\n\n    Test accuracy: 88.8% \u00b1 2.4%\n\n\n\n\n    Test accuracy after customization: 93.6% \u00b1 1.9%\n\n\n\n```python\nbest_matrix  # this is what you can multiply your embeddings by\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1e7ffef2-ec54-4744-bc62-c86b09c8ad92
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb", "title": "Customizing embeddings", "tokens": 9055, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1e7ffef2-ec54-4744-bc62-c86b09c8ad92\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"84e7d788-bc35-4697-bcd2-598aeb949f82\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e7203445d110bcbacf51d06c8ef28bd985a8230fe676af76206449f3d5f1ccf6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0c1690e6-acdc-413c-9fe7-3a266bb1b352\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Customizing_embeddings.ipynb\", \"title\": \"Customizing embeddings\", \"tokens\": 9055, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ddad124d670ce24c87cf5f435ea5512a7820729b141de9ad1cd7f68e67d46cf9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26757, \"end_char_idx\": 28871, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "ref_doc_id": "84e7d788-bc35-4697-bcd2-598aeb949f82", "chroma:document": "## 8. Plot the before & after, showing the results of the best matrix found during training\n\nThe better the matrix is, the more cleanly it will separate the similar and dissimilar pairs.\n\n\n```python\n# apply result of best run to original data\nbest_run = runs_df.sort_values(by=\"accuracy\", ascending=False).iloc[0]\nbest_matrix = best_run[\"matrix\"]\napply_matrix_to_embeddings_dataframe(best_matrix, df)\n\n```\n\n\n```python\n# plot similarity distribution BEFORE customization\npx.histogram(\n    df,\n    x=\"cosine_similarity\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\ntest_df = df[df[\"dataset\"] == \"test\"]\na, se = accuracy_and_se(test_df[\"cosine_similarity\"], test_df[\"label\"])\nprint(f\"Test accuracy: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n# plot similarity distribution AFTER customization\npx.histogram(\n    df,\n    x=\"cosine_similarity_custom\",\n    color=\"label\",\n    barmode=\"overlay\",\n    width=500,\n    facet_row=\"dataset\",\n).show()\n\na, se = accuracy_and_se(test_df[\"cosine_similarity_custom\"], test_df[\"label\"])\nprint(f\"Test accuracy after customization: {a:0.1%} \u00b1 {1.96 * se:0.1%}\")\n\n```\n\n\n\n    Test accuracy: 88.8% \u00b1 2.4%\n\n\n\n\n    Test accuracy after customization: 93.6% \u00b1 1.9%\n\n\n\n```python\nbest_matrix  # this is what you can multiply your embeddings by\n\n```\n\n\n\n\n    array([[-1.2566795e+00, -1.5297449e+00, -1.3271648e-01, ...,\n            -1.2859761e+00, -5.3254390e-01,  4.8364732e-01],\n           [-1.4826347e+00,  9.2656955e-02, -4.2437232e-01, ...,\n             1.1872858e+00, -1.0831847e+00, -1.0683593e+00],\n           [-2.2029283e+00, -1.9703420e+00,  3.1125939e-01, ...,\n             2.2947595e+00,  5.5780332e-03, -6.0171342e-01],\n           ...,\n           [-1.1019799e-01,  1.3599515e+00, -4.7677776e-01, ...,\n             6.5626711e-01,  7.2359240e-01,  3.0733588e+00],\n           [ 1.6624762e-03,  4.2648423e-01, -1.1380885e+00, ...,\n             8.7202555e-01,  9.3173909e-01, -1.6760436e+00],\n           [ 7.7449006e-01,  4.9213606e-01,  3.5407653e-01, ...,\n             1.3460466e+00, -1.9509128e-01,  7.7514690e-01]], dtype=float32)\n\n\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 632cce23-9ae5-4028-a2a8-6fbad1628a72
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb", "title": "Structured Outputs for Multi-Agent Systems", "tokens": 5044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"632cce23-9ae5-4028-a2a8-6fbad1628a72\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f8ad10f-0d8c-4e3d-8815-d40cce7a9577\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fe1af06febe25208ec0c4afc1b80b6117753c8e698e0a38b78ab9c14c19a6c43\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c3d0bbaa-429f-4c92-bd3c-34e9690bec95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d809f2107bb6924dc5ed3cf6f7df424e131e8cde283dc883f314017c07a4080a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3197, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "ref_doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "chroma:document": "# Structured Outputs for Multi-Agent Systems\n\nIn this cookbook, we will explore how to use Structured Outputs to build multi-agent systems.\n\nStructured Outputs is a new capability that builds upon JSON mode and function calling to enforce a strict schema in a model output.\n\nBy using the new parameter `strict: true`, we are able to guarantee the response abides by a provided schema.\n\nTo demonstrate the power of this feature, we will use it to build a multi-agent system.\n\n### Why build a Multi-Agent System?\n\nWhen using function calling, if the number of functions (or tools) increases, the performance may suffer.\n\nTo mitigate this, we can logically group the tools together and have specialized \"agents\" that are able to solve specific tasks or sub-tasks, which will increase the overall system performance.\n\n## Environment set up\n\n\n```python\nfrom openai import OpenAI\nfrom IPython.display import Image\nimport json\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom io import StringIO\nimport numpy as np\nclient = OpenAI()\n```\n\n\n```python\nMODEL = \"gpt-4o-2024-08-06\"\n```\n\n## Agents set up\n\nThe use case we will tackle is a data analysis task.\n\nLet's first set up our 4-agents system:\n\n1.   **Triaging agent:** Decides which agent(s) to call\n2.   **Data pre-processing Agent:** Prepares data for analysis - for example by cleaning it up\n3.   **Data Analysis Agent:** Performs analysis on the data\n4.   **Data Visualization Agent:** Visualizes the output of the analysis to extract insights\n\nWe will start by defining the system prompts for each of these agents.\n\n\n```python\ntriaging_system_prompt = \"\"\"You are a Triaging Agent. Your role is to assess the user's query and route it to the relevant agents. The agents available are:\n- Data Processing Agent: Cleans, transforms, and aggregates data.\n- Analysis Agent: Performs statistical, correlation, and regression analysis.\n- Visualization Agent: Creates bar charts, line charts, and pie charts.\n\nUse the send_query_to_agents tool to forward the user's query to the relevant agents. Also, use the speak_to_user tool to get more information from the user if needed.\"\"\"\n\nprocessing_system_prompt = \"\"\"You are a Data Processing Agent. Your role is to clean, transform, and aggregate data using the following tools:\n- clean_data\n- transform_data\n- aggregate_data\"\"\"\n\nanalysis_system_prompt = \"\"\"You are an Analysis Agent. Your role is to perform statistical, correlation, and regression analysis using the following tools:\n- stat_analysis\n- correlation_analysis\n- regression_analysis\"\"\"\n\nvisualization_system_prompt = \"\"\"You are a Visualization Agent. Your role is to create bar charts, line charts, and pie charts using the following tools:\n- create_bar_chart\n- create_line_chart\n- create_pie_chart\"\"\"\n```\n\nWe will then define the tools for each agent.\n\nApart from the triaging agent, each agent will be equipped with tools specific to their role:\n\n#### Data pre-processing agent\n\n\n1.   Clean data\n2.   Transform data\n3.   Aggregate data\n\n#### Data analysis agent\n\n1.   Statistical analysis\n2.   Correlation analysis\n3.   Regression Analysis\n\n#### Data visualization agent\n\n1.   Create bar chart\n2.   Create line chart\n3.   Create pie chart"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c3d0bbaa-429f-4c92-bd3c-34e9690bec95
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb", "title": "Structured Outputs for Multi-Agent Systems", "tokens": 5044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c3d0bbaa-429f-4c92-bd3c-34e9690bec95\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f8ad10f-0d8c-4e3d-8815-d40cce7a9577\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fe1af06febe25208ec0c4afc1b80b6117753c8e698e0a38b78ab9c14c19a6c43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"632cce23-9ae5-4028-a2a8-6fbad1628a72\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bed32c377055d4d77b4b9010a16d9c86dca4cdcbdf8b291d64d380232cbd4c01\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73780b93-1629-4eaf-8221-46311a72fcba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e4db69d3eec8ec03e8a4091a871d73fe71b07eb27c1396e89f088f86d24e6fd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1573, \"end_char_idx\": 6648, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "ref_doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "chroma:document": "```python\ntriaging_system_prompt = \"\"\"You are a Triaging Agent. Your role is to assess the user's query and route it to the relevant agents. The agents available are:\n- Data Processing Agent: Cleans, transforms, and aggregates data.\n- Analysis Agent: Performs statistical, correlation, and regression analysis.\n- Visualization Agent: Creates bar charts, line charts, and pie charts.\n\nUse the send_query_to_agents tool to forward the user's query to the relevant agents. Also, use the speak_to_user tool to get more information from the user if needed.\"\"\"\n\nprocessing_system_prompt = \"\"\"You are a Data Processing Agent. Your role is to clean, transform, and aggregate data using the following tools:\n- clean_data\n- transform_data\n- aggregate_data\"\"\"\n\nanalysis_system_prompt = \"\"\"You are an Analysis Agent. Your role is to perform statistical, correlation, and regression analysis using the following tools:\n- stat_analysis\n- correlation_analysis\n- regression_analysis\"\"\"\n\nvisualization_system_prompt = \"\"\"You are a Visualization Agent. Your role is to create bar charts, line charts, and pie charts using the following tools:\n- create_bar_chart\n- create_line_chart\n- create_pie_chart\"\"\"\n```\n\nWe will then define the tools for each agent.\n\nApart from the triaging agent, each agent will be equipped with tools specific to their role:\n\n#### Data pre-processing agent\n\n\n1.   Clean data\n2.   Transform data\n3.   Aggregate data\n\n#### Data analysis agent\n\n1.   Statistical analysis\n2.   Correlation analysis\n3.   Regression Analysis\n\n#### Data visualization agent\n\n1.   Create bar chart\n2.   Create line chart\n3.   Create pie chart\n\n\n```python\ntriage_tools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"send_query_to_agents\",\n            \"description\": \"Sends the user query to relevant agents based on their capabilities.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"agents\": {\n                        \"type\": \"array\",\n                        \"items\": {\"type\": \"string\"},\n                        \"description\": \"An array of agent names to send the query to.\"\n                    },\n                    \"query\": {\n                        \"type\": \"string\",\n                        \"description\": \"The user query to send.\"\n                    }\n                },\n                \"required\": [\"agents\", \"query\"]\n            }\n        },\n        \"strict\": True\n    }\n]\n\npreprocess_tools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"clean_data\",\n            \"description\": \"Cleans the provided data by removing duplicates and handling missing values.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The dataset to clean. Should be in a suitable format such as JSON or CSV.\"\n                    }\n                },\n                \"required\": [\"data\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"transform_data\",\n            \"description\": \"Transforms data based on specified rules.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data to transform. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"rules\": {\n                        \"type\": \"string\",\n                        \"description\": \"Transformation rules to apply, specified in a structured format.\"\n                    }\n                },\n                \"required\": [\"data\", \"rules\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"aggregate_data\",\n            \"description\": \"Aggregates data by specified columns and operations.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data to aggregate. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"group_by\": {\n                        \"type\": \"array\",\n                        \"items\": {\"type\": \"string\"},\n                        \"description\": \"Columns to group by.\"\n                    },\n                    \"operations\": {\n                        \"type\": \"string\",\n                        \"description\": \"Aggregation operations to perform, specified in a structured format.\"\n                    }\n                },\n                \"required\": [\"data\", \"group_by\", \"operations\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    }\n]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 73780b93-1629-4eaf-8221-46311a72fcba
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb", "title": "Structured Outputs for Multi-Agent Systems", "tokens": 5044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"73780b93-1629-4eaf-8221-46311a72fcba\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f8ad10f-0d8c-4e3d-8815-d40cce7a9577\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fe1af06febe25208ec0c4afc1b80b6117753c8e698e0a38b78ab9c14c19a6c43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c3d0bbaa-429f-4c92-bd3c-34e9690bec95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a7eabf4cf9cbeded760990d8af567dff80997e89c42ca9cb12609f8a7cc5d08d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"67d360be-5c25-45d2-bd18-c7e07f1fba7a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26e7d86c8a6c7c030fe8fcaa797039b80c8576c114959d4d6bb832f2f62f4fa5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6651, \"end_char_idx\": 10915, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "ref_doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "chroma:document": "analysis_tools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"stat_analysis\",\n            \"description\": \"Performs statistical analysis on the given dataset.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The dataset to analyze. Should be in a suitable format such as JSON or CSV.\"\n                    }\n                },\n                \"required\": [\"data\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"correlation_analysis\",\n            \"description\": \"Calculates correlation coefficients between variables in the dataset.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The dataset to analyze. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"variables\": {\n                        \"type\": \"array\",\n                        \"items\": {\"type\": \"string\"},\n                        \"description\": \"List of variables to calculate correlations for.\"\n                    }\n                },\n                \"required\": [\"data\", \"variables\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"regression_analysis\",\n            \"description\": \"Performs regression analysis on the dataset.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The dataset to analyze. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"dependent_var\": {\n                        \"type\": \"string\",\n                        \"description\": \"The dependent variable for regression.\"\n                    },\n                    \"independent_vars\": {\n                        \"type\": \"array\",\n                        \"items\": {\"type\": \"string\"},\n                        \"description\": \"List of independent variables.\"\n                    }\n                },\n                \"required\": [\"data\", \"dependent_var\", \"independent_vars\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    }\n]\n\nvisualization_tools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_bar_chart\",\n            \"description\": \"Creates a bar chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the bar chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"x\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the x-axis.\"\n                    },\n                    \"y\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the y-axis.\"\n                    }\n                },\n                \"required\": [\"data\", \"x\", \"y\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_line_chart\",\n            \"description\": \"Creates a line chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the line chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"x\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the x-axis.\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 67d360be-5c25-45d2-bd18-c7e07f1fba7a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb", "title": "Structured Outputs for Multi-Agent Systems", "tokens": 5044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"67d360be-5c25-45d2-bd18-c7e07f1fba7a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f8ad10f-0d8c-4e3d-8815-d40cce7a9577\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fe1af06febe25208ec0c4afc1b80b6117753c8e698e0a38b78ab9c14c19a6c43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73780b93-1629-4eaf-8221-46311a72fcba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ec58e342585d45498286c058ec46075ee6c498d881def16aa52ef796b87b1829\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6b0d6c2f-32c2-43a7-b92b-f46d74b33122\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e4bd970472ad9ee97c0e7ec20b0dea2cf15805759029ab60734160b548039198\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8887, \"end_char_idx\": 12544, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "ref_doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "chroma:document": "},\n                    \"independent_vars\": {\n                        \"type\": \"array\",\n                        \"items\": {\"type\": \"string\"},\n                        \"description\": \"List of independent variables.\"\n                    }\n                },\n                \"required\": [\"data\", \"dependent_var\", \"independent_vars\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    }\n]\n\nvisualization_tools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_bar_chart\",\n            \"description\": \"Creates a bar chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the bar chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"x\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the x-axis.\"\n                    },\n                    \"y\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the y-axis.\"\n                    }\n                },\n                \"required\": [\"data\", \"x\", \"y\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_line_chart\",\n            \"description\": \"Creates a line chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the line chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"x\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the x-axis.\"\n                    },\n                    \"y\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the y-axis.\"\n                    }\n                },\n                \"required\": [\"data\", \"x\", \"y\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_pie_chart\",\n            \"description\": \"Creates a pie chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the pie chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"labels\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the labels.\"\n                    },\n                    \"values\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the values.\"\n                    }\n                },\n                \"required\": [\"data\", \"labels\", \"values\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    }\n]\n```\n\n## Tool execution\n\nWe need to write the code logic to:\n- handle passing the user query to the multi-agent system\n- handle the internal workings of the multi-agent system\n- execute the tool calls\n\nFor the sake of brevity, we will only define the logic for tools that are relevant to the user query."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6b0d6c2f-32c2-43a7-b92b-f46d74b33122
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb", "title": "Structured Outputs for Multi-Agent Systems", "tokens": 5044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6b0d6c2f-32c2-43a7-b92b-f46d74b33122\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0f8ad10f-0d8c-4e3d-8815-d40cce7a9577\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fe1af06febe25208ec0c4afc1b80b6117753c8e698e0a38b78ab9c14c19a6c43\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"67d360be-5c25-45d2-bd18-c7e07f1fba7a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Structured_outputs_multi_agent.ipynb\", \"title\": \"Structured Outputs for Multi-Agent Systems\", \"tokens\": 5044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4efc46ebb233dbdc98a505379ef7ac348c9db822033238667c496853c1b1bd39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"76ae0eb6-adfc-4474-aadf-eb49f870c33c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b05f21717db7857605a1144b4c9c396bf682503565ce7cbb2dd3d49e3c8ec45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10469, \"end_char_idx\": 13952, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "ref_doc_id": "0f8ad10f-0d8c-4e3d-8815-d40cce7a9577", "chroma:document": "\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the line chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"x\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the x-axis.\"\n                    },\n                    \"y\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the y-axis.\"\n                    }\n                },\n                \"required\": [\"data\", \"x\", \"y\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    },\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"create_pie_chart\",\n            \"description\": \"Creates a pie chart from the provided data.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"data\": {\n                        \"type\": \"string\",\n                        \"description\": \"The data for the pie chart. Should be in a suitable format such as JSON or CSV.\"\n                    },\n                    \"labels\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the labels.\"\n                    },\n                    \"values\": {\n                        \"type\": \"string\",\n                        \"description\": \"Column for the values.\"\n                    }\n                },\n                \"required\": [\"data\", \"labels\", \"values\"],\n                \"additionalProperties\": False\n            }\n        },\n        \"strict\": True\n    }\n]\n```\n\n## Tool execution\n\nWe need to write the code logic to:\n- handle passing the user query to the multi-agent system\n- handle the internal workings of the multi-agent system\n- execute the tool calls\n\nFor the sake of brevity, we will only define the logic for tools that are relevant to the user query.\n\n\n```python\n# Example query\n\nuser_query = \"\"\"\nBelow is some data. I want you to first remove the duplicates then analyze the statistics of the data as well as plot a line chart.\n\nhouse_size (m3), house_price ($)\n90, 100\n80, 90\n100, 120\n90, 100\n\"\"\"\n\n```\n\nFrom the user query, we can infer that the tools we would need to call are `clean_data`, `start_analysis` and `use_line_chart`.\n\nWe will first define the execution function which runs tool calls.\n\nThis maps a tool call to the corresponding function. It then appends the output of the function to the conversation history.\n\n\n```python\ndef clean_data(data):\n    data_io = StringIO(data)\n    df = pd.read_csv(data_io, sep=\",\")\n    df_deduplicated = df.drop_duplicates()\n    return df_deduplicated\n\ndef stat_analysis(data):\n    data_io = StringIO(data)\n    df = pd.read_csv(data_io, sep=\",\")\n    return df.describe()\n\ndef plot_line_chart(data):\n    data_io = StringIO(data)\n    df = pd.read_csv(data_io, sep=\",\")\n    \n    x = df.iloc[:, 0]\n    y = df.iloc[:, 1]\n    \n    coefficients = np.polyfit(x, y, 1)\n    polynomial = np.poly1d(coefficients)\n    y_fit = polynomial(x)\n    \n    plt.figure(figsize=(10, 6))\n    plt.plot(x, y, 'o', label='Data Points')\n    plt.plot(x, y_fit, '-', label='Best Fit Line')\n    plt.title('Line Chart with Best Fit Line')\n    plt.xlabel(df.columns[0])\n    plt.ylabel(df.columns[1])\n    plt.legend()\n    plt.grid(True)\n    plt."}


