# embedding_metadata

key: chroma:document
string_value:
However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.
    
    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.
    
    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.
    
    The concept of "Friendly AI" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.
    
    Other approaches to AI ethics include Wendell Wallach's concept of "artificial moral agents" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.
    
    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.
    
    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.


key: _node_content
string_value:
{"id_": "11926555-d728-4c09-9c7f-e627d2a1f1f1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "22cf6d2f-4b51-4122-b1e1-f29aaccba528", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c59eca7ae72f7b566c24c3cb86b377e227ee7c83df822cefd54c90ea90fbf16f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3e7efa5-a35e-4782-8841-3bb0ddf4db05", "node_type": "1", "metadata": {}, "hash": "11840109df54a1b0d934e0054aaa8d15461fc64385f52be2a7382fa7ce8cd065", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35491, "end_char_idx": 39409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.
    
    The concept of "Friendly AI" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.
    
    Other approaches to AI ethics include Wendell Wallach's concept of "artificial moral agents" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.
    
    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.
    
    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.
    
    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.
    
    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.
    
    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.
    
    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an "electronic brain."
    
    Early research in artificial intelligence (AI) included the development of "artificial neurons" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence.


key: _node_content
string_value:
{"id_": "c3e7efa5-a35e-4782-8841-3bb0ddf4db05", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "11926555-d728-4c09-9c7f-e627d2a1f1f1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "59ab7471190a82b9586d59abe2fb1c194d710a582e0323d2c686b661ba2e7046", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad", "node_type": "1", "metadata": {}, "hash": "a573034e26b7f0acda81e701e0c415f8e5577ee3a3e300f41ce293816f815a69", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37475, "end_char_idx": 41372, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.
    
    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.
    
    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.
    
    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.
    
    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an "electronic brain."
    
    Early research in artificial intelligence (AI) included the development of "artificial neurons" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence. The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.
    
    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the "AI winter" where funding was scarce.
    
    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.
    
    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits.


key: _node_content
string_value:
{"id_": "b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3e7efa5-a35e-4782-8841-3bb0ddf4db05", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2c4b611156ecb41838a6edc8be145e62ef91bbb675f22c1a8c0fdfccf3fb69fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7379fe50-66b1-4f1b-8a0d-a2c2a84bc622", "node_type": "1", "metadata": {}, "hash": "13f2c9a2c3e9446172b0a79a8a36d81a9cc9f6f21555340cdf7b9c121750b821", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39410, "end_char_idx": 43068, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.
    
    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the "AI winter" where funding was scarce.
    
    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.
    
    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits.
    
    AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.
    
    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.
    
    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.
    
    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved.


key: _node_content
string_value:
{"id_": "7379fe50-66b1-4f1b-8a0d-a2c2a84bc622", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0b799772952f1e90ecb951eee3f88623fb22464fe203e5a0e1d6c7c208bbc8bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aaa5acd2-cfc6-4ef3-8913-70466af670a1", "node_type": "1", "metadata": {}, "hash": "864ec1be43f33832f22eb87518769abe3da151388b4c9d94758e1ddab55af094", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41382, "end_char_idx": 45201, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.
    
    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.
    
    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.
    
    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved. Alan Turing, in 1950, suggested shifting the focus from whether machines can think to whether they can exhibit intelligent behavior, as demonstrated by his Turing test, which assesses a machine's ability to simulate human conversation. Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.
    
    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.
    
    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.
    
    Symbolic AI, or "GOFAI," focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.
    
    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes.


key: _node_content
string_value:
{"id_": "aaa5acd2-cfc6-4ef3-8913-70466af670a1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7379fe50-66b1-4f1b-8a0d-a2c2a84bc622", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "813b60ab99168799a13ae97f607b1b5200b5bc4d9d1e288056dcc3eab247b42e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "582e3747-b7b8-42bc-a1f2-09a41a42fea0", "node_type": "1", "metadata": {}, "hash": "581edbf37bf65be3eca384998eb42a7a7b72f7e2bdb5182dc8ec362f5d914196", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43305, "end_char_idx": 47330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.
    
    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.
    
    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.
    
    Symbolic AI, or "GOFAI," focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.
    
    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes. This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.
    
    In AI development, there has been a historical division between "Neats," who believe intelligent behavior can be described with simple principles, and "Scruffies," who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.
    
    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.
    
    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.
    
    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.
    
    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the "hard" problem of consciousness, which is understanding why or how brain processes feel like something, and the "easy" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain.


key: _node_content
string_value:
{"id_": "582e3747-b7b8-42bc-a1f2-09a41a42fea0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aaa5acd2-cfc6-4ef3-8913-70466af670a1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efb5a6dca58996ed87446b03e31ade84f0a1d5c8dbe1afc8a110e54d4fe0b087", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66ad818f-443e-4a43-beb3-a6962e4a228f", "node_type": "1", "metadata": {}, "hash": "398774b8edc77695f25fa8e430e5a28985bf15c9744e4117a0857105aae65ca5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45202, "end_char_idx": 49352, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.
    
    In AI development, there has been a historical division between "Neats," who believe intelligent behavior can be described with simple principles, and "Scruffies," who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.
    
    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.
    
    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.
    
    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.
    
    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the "hard" problem of consciousness, which is understanding why or how brain processes feel like something, and the "easy" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain.
    
    In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.
    
    The concept of "strong AI," as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting "electronic personhood" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.
    
    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.
    
    The concept of an "intelligence explosion" or "singularity" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger.


key: _node_content
string_value:
{"id_": "66ad818f-443e-4a43-beb3-a6962e4a228f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "582e3747-b7b8-42bc-a1f2-09a41a42fea0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ad2f637a8d75ab1bb12403da5652de7b91aab45754e3ea9a6c1811d755e1753c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d89c9cd-fff4-4dd5-8188-f425ed8fb30a", "node_type": "1", "metadata": {}, "hash": "735202f5d0d163c480984ba0dbbac7522e5875733118a19e29b5863de67afa79", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 47340, "end_char_idx": 51070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.
    
    The concept of "strong AI," as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting "electronic personhood" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.
    
    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.
    
    The concept of an "intelligence explosion" or "singularity" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.
    
    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term "robot" was first introduced by Karel Čapek in 1921. Notable narratives include Mary Shelley's "Frankenstein" and films like "2001: A Space Odyssey" and "The Terminator," which typically showcase AI as a threat. Conversely, loyal robots like Gort from "The Day the Earth Stood Still" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.
    
    Numerous works, including Karel Čapek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.


Note that this utility also allows passing additional instructions.


```python
summary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,
                                                 additional_instructions="Write in point form and focus on numerical data.")
print(summary_with_additional_instructions)
```

    100%|██████████| 5/5 [00:38<00:00,  7.73s/it]

    - AI is intelligence demonstrated by machines, especially computer systems.


key: _node_content
string_value:
{"id_": "0d89c9cd-fff4-4dd5-8188-f425ed8fb30a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66ad818f-443e-4a43-beb3-a6962e4a228f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "08903f2e07e2ab2705125126d62572c6d3a1fd6be6cacf5d945af6cec12ac066", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "85c9225d-3fe5-47ab-ace4-d24236caa327", "node_type": "1", "metadata": {}, "hash": "b6f7cd9a363899443e8bb9fede99ebb1981ee51391e798233b91d5e0d676868c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49268, "end_char_idx": 53081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.
    
    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term "robot" was first introduced by Karel Čapek in 1921. Notable narratives include Mary Shelley's "Frankenstein" and films like "2001: A Space Odyssey" and "The Terminator," which typically showcase AI as a threat. Conversely, loyal robots like Gort from "The Day the Earth Stood Still" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.
    
    Numerous works, including Karel Čapek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.


Note that this utility also allows passing additional instructions.


```python
summary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,
                                                 additional_instructions="Write in point form and focus on numerical data.")
print(summary_with_additional_instructions)
```

    100%|██████████| 5/5 [00:38<00:00,  7.73s/it]

    - AI is intelligence demonstrated by machines, especially computer systems.
    - AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.
    - Alan Turing initiated substantial AI research, termed "machine intelligence."
    - AI became an academic discipline in 1956, experiencing cycles of optimism and "AI winters."
    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.
    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.
    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.
    - AI techniques include search, optimization, logic, neural networks, and statistical methods.
    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.
    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.
    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.
    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.
    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.
    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.
    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.
    - General AI aims to solve diverse problems with human-like versatility.
    - AI search techniques include state space search, local search, and adversarial search for game-playing.
    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation.


key: _node_content
string_value:
{"id_": "85c9225d-3fe5-47ab-ace4-d24236caa327", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d89c9cd-fff4-4dd5-8188-f425ed8fb30a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d5453c42c31d0447eaa16744b9861eedf55beb20ef035c244cf1281da5f823f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4d40f8e-34de-4021-8e92-5603bd674c4b", "node_type": "1", "metadata": {}, "hash": "2251f73f36332b9b5dd7e614be2e7eea5e64ccfe22a834d835dbe0496f2b2e33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51075, "end_char_idx": 55070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.
    - Alan Turing initiated substantial AI research, termed "machine intelligence."
    - AI became an academic discipline in 1956, experiencing cycles of optimism and "AI winters."
    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.
    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.
    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.
    - AI techniques include search, optimization, logic, neural networks, and statistical methods.
    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.
    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.
    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.
    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.
    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.
    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.
    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.
    - General AI aims to solve diverse problems with human-like versatility.
    - AI search techniques include state space search, local search, and adversarial search for game-playing.
    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation.
    - Probabilistic methods in AI address decision-making and planning under uncertainty using tools like Bayesian networks and Markov decision processes.
    - Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.
    
    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.
    - Deep neural networks: At least 2 hidden layers.
    - Training techniques: Commonly use backpropagation.
    - Feedforward networks: Signal passes in one direction.
    - Recurrent networks: Output fed back into input for short-term memory.
    - Perceptrons: Single layer of neurons.
    - Convolutional networks: Strengthen connections between close neurons, important in image processing.
    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.
    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.
    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.
    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.
    - AI in healthcare: Increases patient care, used in medical research and drug discovery.
    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.
    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.
    - Generative AI: Creates realistic images and texts, used in creative arts.
    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.
    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.
    
    - AI engineers demographics: 4% black, 20% women.
    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias.


key: _node_content
string_value:
{"id_": "e4d40f8e-34de-4021-8e92-5603bd674c4b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85c9225d-3fe5-47ab-ace4-d24236caa327", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7acafa82a61d335e6c39aa2ff693c54d5a018ac485002455800a1c6ddea20e1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e73abe5-4112-4981-be1a-87f7c19b47e7", "node_type": "1", "metadata": {}, "hash": "722d5d71b192396b7060fdd300c835f901960f6e116f43af0787b9b06cb0c5df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53241, "end_char_idx": 56938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.
    
    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.
    - Deep neural networks: At least 2 hidden layers.
    - Training techniques: Commonly use backpropagation.
    - Feedforward networks: Signal passes in one direction.
    - Recurrent networks: Output fed back into input for short-term memory.
    - Perceptrons: Single layer of neurons.
    - Convolutional networks: Strengthen connections between close neurons, important in image processing.
    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.
    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.
    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.
    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.
    - AI in healthcare: Increases patient care, used in medical research and drug discovery.
    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.
    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.
    - Generative AI: Creates realistic images and texts, used in creative arts.
    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.
    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.
    
    - AI engineers demographics: 4% black, 20% women.
    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias.
    - AI complexity: Designers often can't explain decision-making processes.
    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as "cancerous"; AI misclassifies asthma patients as low risk for pneumonia.
    - Right to explanation: Essential for accountability, especially in medical and legal fields.
    - DARPA's XAI program (2014): Aims to make AI decisions understandable.
    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.
    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.
    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.
    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.
    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.
    - Ethical AI development: Importance of aligning AI with human values and ethics.
    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.
    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.
    
    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.
    - 1985: AI market value exceeded $1 billion.
    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.
    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.


key: _node_content
string_value:
{"id_": "6e73abe5-4112-4981-be1a-87f7c19b47e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4d40f8e-34de-4021-8e92-5603bd674c4b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "97b4da39108479dbdaccdc74ea4140b45cf8f555a10f856b99977317d20270fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347", "node_type": "1", "metadata": {}, "hash": "4836df9d040963cf0e16b9da90dd973ca629a7122f0bc4f12d13376023401dd0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55075, "end_char_idx": 58470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- AI complexity: Designers often can't explain decision-making processes.
    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as "cancerous"; AI misclassifies asthma patients as low risk for pneumonia.
    - Right to explanation: Essential for accountability, especially in medical and legal fields.
    - DARPA's XAI program (2014): Aims to make AI decisions understandable.
    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.
    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.
    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.
    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.
    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.
    - Ethical AI development: Importance of aligning AI with human values and ethics.
    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.
    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.
    
    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.
    - 1985: AI market value exceeded $1 billion.
    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.
    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.
    - 2016: Fairness and misuse of technology became central issues in AI.
    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.
    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.
    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.
    - 2010s: Statistical machine learning overshadowed other AI approaches.
    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.
    - Late 1980s: Introduction of soft computing techniques.
    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).
    - 2017: EU considered granting "electronic personhood" to advanced AI systems.
    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.
    
    - Focus on how AI and technology, as depicted in "Ex Machina" and Philip K. Dick's "Do Androids Dream of Electric Sheep?", alter human subjectivity.
    - No specific numerical data provided.


    


Finally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary.


key: _node_content
string_value:
{"id_": "d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e73abe5-4112-4981-be1a-87f7c19b47e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1272d25b95df50690896a0bbe6b7e2467661491b69e238e0a00527c51bb69679", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "622fb7e0-a21f-4cf4-9bae-4250afd14342", "node_type": "1", "metadata": {}, "hash": "6af23db3cfada4837d1c9eda7bfb3cfeb5b39b6719ced843d38bf788adb5e193", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 56596, "end_char_idx": 60318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.
    - 2016: Fairness and misuse of technology became central issues in AI.
    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.
    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.
    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.
    - 2010s: Statistical machine learning overshadowed other AI approaches.
    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.
    - Late 1980s: Introduction of soft computing techniques.
    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).
    - 2017: EU considered granting "electronic personhood" to advanced AI systems.
    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.
    
    - Focus on how AI and technology, as depicted in "Ex Machina" and Philip K. Dick's "Do Androids Dream of Electric Sheep?", alter human subjectivity.
    - No specific numerical data provided.


    


Finally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary.


```python
recursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)
print(recursive_summary)
```

    100%|██████████| 5/5 [00:41<00:00,  8.36s/it]

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.
    
    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.
    
    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles.


key: _node_content
string_value:
{"id_": "622fb7e0-a21f-4cf4-9bae-4250afd14342", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "28f6cf44b6dfc6e07ceb7ca021cf307a49a6a4aedacb7898221b3e1685e78f18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04a4bc23-fa70-4f99-82df-88c3eb6b6cca", "node_type": "1", "metadata": {}, "hash": "83289f7789368dd6ba4ebb66db50f7173e758ea8f9bd7d035c073e370feecc61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 58473, "end_char_idx": 62711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
recursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)
print(recursive_summary)
```

    100%|██████████| 5/5 [00:41<00:00,  8.36s/it]

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.
    
    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.
    
    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles. AI research, which began as a distinct academic discipline in 1956, includes sub-fields like natural language processing and robotics, employing techniques from various scientific domains. AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.
    
    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.
    
    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.
    
    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as "AI winters." Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.
    
    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.
    
    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks.


key: _node_content
string_value:
{"id_": "04a4bc23-fa70-4f99-82df-88c3eb6b6cca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "622fb7e0-a21f-4cf4-9bae-4250afd14342", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d23431ff39f20557d75f09d4eea04ad762c6ba893c72bf4034627c1967185577", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "538a5c65-1597-4d68-b0d9-247e559afe00", "node_type": "1", "metadata": {}, "hash": "9a866023518cbeefe2618e8ba1ba996eacd2f57fe8a01ef1a660743355cab8a4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 60508, "end_char_idx": 64818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.
    
    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.
    
    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.
    
    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as "AI winters." Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.
    
    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.
    
    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks. AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.
    
    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.
    
    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first "AI winter." The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring "sub-symbolic" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun’s convolutional neural networks for digit recognition.
    
    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.
    
    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI.


key: _node_content
string_value:
{"id_": "538a5c65-1597-4d68-b0d9-247e559afe00", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04a4bc23-fa70-4f99-82df-88c3eb6b6cca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "941546bc1b255a66a985801db6cf8b82f90e33d9fea6315f187d0e3a6f84f5ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 62712, "end_char_idx": 65489, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.
    
    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.
    
    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first "AI winter." The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring "sub-symbolic" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun’s convolutional neural networks for digit recognition.
    
    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.
    
    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI. The concept of AI rights and welfare has also emerged, reflecting concerns about the moral status of advanced AI systems.
    
    Overall, AI research has oscillated between periods of intense optimism and profound setbacks, with current trends heavily favoring practical applications through narrow AI, while continuing to explore the broader implications and potential of general and superintelligent AI systems.
    
    Artificial Intelligence (AI) and its portrayal in media, such as the film "Ex Machina" and Philip K. Dick's novel "Do Androids Dream of Electric Sheep?", explore how technology, particularly AI, can alter our understanding of human subjectivity.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb


key: _node_content
string_value:
{"id_": "1e8ebf8d-d95c-4f15-9feb-5c4729395286", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c298d6f-a68c-4f31-8644-06aaec1e0b1c", "node_type": "1", "metadata": {}, "hash": "06cc2ffe4a15ae4a93be1e8fe4717b9365c963fe9d70d2f2824f74c62e5c1b13", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3392, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use functions with a knowledge base

This notebook builds on the concepts in the [argument generation](How_to_call_functions_with_chat_models.ipynb) notebook, by creating an agent with access to a knowledge base and two functions that it can call based on the user requirement.

We'll create an agent that uses data from arXiv to answer questions about academic subjects. It has two functions at its disposal:
- **get_articles**: A function that gets arXiv articles on a subject and summarizes them for the user with links.
- **read_article_and_summarize**: This function takes one of the previously searched articles, reads it in its entirety and summarizes the core argument, evidence and conclusions.

This will get you comfortable with a multi-function workflow that can choose from multiple services, and where some of the data from the first function is persisted to be used by the second.

## Walkthrough

This cookbook takes you through the following workflow:

- **Search utilities:** Creating the two functions that access arXiv for answers.
- **Configure Agent:** Building up the Agent behaviour that will assess the need for a function and, if one is required, call that function and present results back to the agent.
- **arXiv conversation:** Put all of this together in live conversation.



```python
!pip install scipy --quiet
!pip install tenacity --quiet
!pip install tiktoken==0.3.3 --quiet
!pip install termcolor --quiet
!pip install openai --quiet
!pip install arxiv --quiet
!pip install pandas --quiet
!pip install PyPDF2 --quiet
!pip install tqdm --quiet
```


```python
import os
import arxiv
import ast
import concurrent
import json
import os
import pandas as pd
import tiktoken
from csv import writer
from IPython.display import display, Markdown, Latex
from openai import OpenAI
from PyPDF2 import PdfReader
from scipy import spatial
from tenacity import retry, wait_random_exponential, stop_after_attempt
from tqdm import tqdm
from termcolor import colored

GPT_MODEL = "gpt-3.5-turbo-0613"
EMBEDDING_MODEL = "text-embedding-ada-002"
client = OpenAI()
```

## Search utilities

We'll first set up some utilities that will underpin our two functions.

Downloaded papers will be stored in a directory (we use ```./data/papers``` here). We create a file ```arxiv_library.csv``` to store the embeddings and details for downloaded papers to retrieve against using ```summarize_text```.


```python
directory = './data/papers'

# Check if the directory already exists
if not os.path.exists(directory):
    # If the directory doesn't exist, create it and any necessary intermediate directories
    os.makedirs(directory)
    print(f"Directory '{directory}' created successfully.")
else:
    # If the directory already exists, print a message indicating it
    print(f"Directory '{directory}' already exists.")
```

    Directory './data/papers' already exists.



```python
# Set a directory to store downloaded papers
data_dir = os.path.join(os.curdir, "data", "papers")
paper_dir_filepath = "./data/arxiv_library.csv"

# Generate a blank dataframe where we can store downloaded files
df = pd.DataFrame(list())
df.to_csv(paper_dir_filepath)
```


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def embedding_request(text):
    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)
    return response


key: _node_content
string_value:
{"id_": "7c298d6f-a68c-4f31-8644-06aaec1e0b1c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1e8ebf8d-d95c-4f15-9feb-5c4729395286", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "faf67a18fc144711c997eb09be7de41baccd8408d0ca6300cc64b34e87a1c239", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd1ea527-b4c9-4ead-861d-8cada290f9c7", "node_type": "1", "metadata": {}, "hash": "eeecb8750b86446247f52017531265dcf093443dfeec836b108fd5c697591a7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2421, "end_char_idx": 5894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
directory = './data/papers'

# Check if the directory already exists
if not os.path.exists(directory):
    # If the directory doesn't exist, create it and any necessary intermediate directories
    os.makedirs(directory)
    print(f"Directory '{directory}' created successfully.")
else:
    # If the directory already exists, print a message indicating it
    print(f"Directory '{directory}' already exists.")
```

    Directory './data/papers' already exists.



```python
# Set a directory to store downloaded papers
data_dir = os.path.join(os.curdir, "data", "papers")
paper_dir_filepath = "./data/arxiv_library.csv"

# Generate a blank dataframe where we can store downloaded files
df = pd.DataFrame(list())
df.to_csv(paper_dir_filepath)
```


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def embedding_request(text):
    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)
    return response


@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def get_articles(query, library=paper_dir_filepath, top_k=5):
    """This function gets the top_k articles based on a user's query, sorted by relevance.
    It also downloads the files and stores them in arxiv_library.csv to be retrieved by the read_article_and_summarize.
    """
    client = arxiv.Client()
    search = arxiv.Search(
        query = "quantum",
        max_results = 10,
        sort_by = arxiv.SortCriterion.SubmittedDate
    )
    result_list = []
    for result in client.results(search):
        result_dict = {}
        result_dict.update({"title": result.title})
        result_dict.update({"summary": result.summary})

        # Taking the first url provided
        result_dict.update({"article_url": [x.href for x in result.links][0]})
        result_dict.update({"pdf_url": [x.href for x in result.links][1]})
        result_list.append(result_dict)

        # Store references in library file
        response = embedding_request(text=result.title)
        file_reference = [
            result.title,
            result.download_pdf(data_dir),
            response.data[0].embedding,
        ]

        # Write to file
        with open(library, "a") as f_object:
            writer_object = writer(f_object)
            writer_object.writerow(file_reference)
            f_object.close()
    return result_list

```


```python
# Test that the search is working
result_output = get_articles("ppo reinforcement learning")
result_output[0]

```




    {'title': 'Quantum types: going beyond qubits and quantum gates',
     'summary': 'Quantum computing is a growing field with significant potential applications.\nLearning how to code quantum programs means understanding how qubits work and\nlearning to use quantum gates. This is analogous to creating classical\nalgorithms using logic gates and bits. Even after learning all concepts, it is\ndifficult to create new algorithms, which hinders the acceptance of quantum\nprogramming by most developers. This article outlines the need for higher-level\nabstractions and proposes some of them in a developer-friendly programming\nlanguage called Rhyme. The new quantum types are extensions of classical types,\nincluding bits, integers, floats, characters, arrays, and strings. We show how\nto use such types with code snippets.',
     'article_url': 'http://arxiv.org/abs/2401.15073v1',
     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}


key: _node_content
string_value:
{"id_": "bd1ea527-b4c9-4ead-861d-8cada290f9c7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c298d6f-a68c-4f31-8644-06aaec1e0b1c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7e02496287c4d245cfb6cc65fb2bf54b867c6d86561811fe3755f11da758fb3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c103e97-4cf9-4391-a550-cf323ca1ec93", "node_type": "1", "metadata": {}, "hash": "7163b04905084568652b4d4061f5447108bd80f5e5f144bcc18056e7382f7dcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4824, "end_char_idx": 7822, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test that the search is working
result_output = get_articles("ppo reinforcement learning")
result_output[0]

```




    {'title': 'Quantum types: going beyond qubits and quantum gates',
     'summary': 'Quantum computing is a growing field with significant potential applications.\nLearning how to code quantum programs means understanding how qubits work and\nlearning to use quantum gates. This is analogous to creating classical\nalgorithms using logic gates and bits. Even after learning all concepts, it is\ndifficult to create new algorithms, which hinders the acceptance of quantum\nprogramming by most developers. This article outlines the need for higher-level\nabstractions and proposes some of them in a developer-friendly programming\nlanguage called Rhyme. The new quantum types are extensions of classical types,\nincluding bits, integers, floats, characters, arrays, and strings. We show how\nto use such types with code snippets.',
     'article_url': 'http://arxiv.org/abs/2401.15073v1',
     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}




```python
def strings_ranked_by_relatedness(
    query: str,
    df: pd.DataFrame,
    relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),
    top_n: int = 100,
) -> list[str]:
    """Returns a list of strings and relatednesses, sorted from most related to least."""
    query_embedding_response = embedding_request(query)
    query_embedding = query_embedding_response.data[0].embedding
    strings_and_relatednesses = [
        (row["filepath"], relatedness_fn(query_embedding, row["embedding"]))
        for i, row in df.iterrows()
    ]
    strings_and_relatednesses.sort(key=lambda x: x[1], reverse=True)
    strings, relatednesses = zip(*strings_and_relatednesses)
    return strings[:top_n]

```


```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


key: _node_content
string_value:
{"id_": "6c103e97-4cf9-4391-a550-cf323ca1ec93", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd1ea527-b4c9-4ead-861d-8cada290f9c7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "66ebd595220fcf8427bec5345ba9961fb2f8eba209bf2a68f1846be54f97c071", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c21350d0-23de-41c9-90d7-5ef0f09dc14b", "node_type": "1", "metadata": {}, "hash": "f8368b1e892457b8dadde4755ca24de7594d075628542d9c89386fa85104d0de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6615, "end_char_idx": 8210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


def extract_chunk(content, template_prompt):
    """This function applies a prompt to some input content. In this case it returns a summarized chunk of text"""
    prompt = template_prompt + content
    response = client.chat.completions.create(
        model=GPT_MODEL, messages=[{"role": "user", "content": prompt}], temperature=0
    )
    return response.choices[0].message.content


key: _node_content
string_value:
{"id_": "c21350d0-23de-41c9-90d7-5ef0f09dc14b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c103e97-4cf9-4391-a550-cf323ca1ec93", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "82e2368b97ff83bebf2e1332c096754ed5324a9487ccdcc17b4c29d006727895", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "631a23d4-4e44-41d6-aca0-40f0b3af43c8", "node_type": "1", "metadata": {}, "hash": "6f17afc756f7302089437434ee8c74203da2c741dae57a6945a3895b22db4707", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6615, "end_char_idx": 11063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


def extract_chunk(content, template_prompt):
    """This function applies a prompt to some input content. In this case it returns a summarized chunk of text"""
    prompt = template_prompt + content
    response = client.chat.completions.create(
        model=GPT_MODEL, messages=[{"role": "user", "content": prompt}], temperature=0
    )
    return response.choices[0].message.content


def summarize_text(query):
    """This function does the following:
    - Reads in the arxiv_library.csv file in including the embeddings
    - Finds the closest file to the user's query
    - Scrapes the text out of the file and chunks it
    - Summarizes each chunk in parallel
    - Does one final summary and returns this to the user"""

    # A prompt to dictate how the recursive summarizations should approach the input paper
    summary_prompt = """Summarize this text from an academic paper. Extract any key points with reasoning.\n\nContent:"""

    # If the library is empty (no searches have been performed yet), we perform one and download the results
    library_df = pd.read_csv(paper_dir_filepath).reset_index()
    if len(library_df) == 0:
        print("No papers searched yet, downloading first.")
        get_articles(query)
        print("Papers downloaded, continuing")
        library_df = pd.read_csv(paper_dir_filepath).reset_index()
    library_df.columns = ["title", "filepath", "embedding"]
    library_df["embedding"] = library_df["embedding"].apply(ast.literal_eval)
    strings = strings_ranked_by_relatedness(query, library_df, top_n=1)
    print("Chunking text from paper")
    pdf_text = read_pdf(strings[0])

    # Initialise tokenizer
    tokenizer = tiktoken.get_encoding("cl100k_base")
    results = ""

    # Chunk up the document into 1500 token chunks
    chunks = create_chunks(pdf_text, 1500, tokenizer)
    text_chunks = [tokenizer.decode(chunk) for chunk in chunks]
    print("Summarizing each chunk of text")

    # Parallel process the summaries
    with concurrent.futures.ThreadPoolExecutor(
        max_workers=len(text_chunks)
    ) as executor:
        futures = [
            executor.submit(extract_chunk, chunk, summary_prompt)
            for chunk in text_chunks
        ]
        with tqdm(total=len(text_chunks)) as pbar:
            for _ in concurrent.futures.as_completed(futures):
                pbar.update(1)
        for future in futures:
            data = future.result()
            results += data

    # Final summary
    print("Summarizing into overall summary")
    response = client.chat.completions.create(
        model=GPT_MODEL,
        messages=[
            {
                "role": "user",
                "content": f"""Write a summary collated from this collection of key points extracted from an academic paper.
                        The summary should highlight the core argument, conclusions and evidence, and answer the user's query.
                        User query: {query}
                        The summary should be structured in bulleted lists following the headings Core Argument, Evidence, and Conclusions.
                        Key points:\n{results}\nSummary:\n""",
            }
        ],
        temperature=0,
    )
    return response

```


key: _node_content
string_value:
{"id_": "631a23d4-4e44-41d6-aca0-40f0b3af43c8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c21350d0-23de-41c9-90d7-5ef0f09dc14b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e3938444d746a767bd2eec1d57e73306a7398c1cb3385ff2aaba09225f3d5110", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "705c20f1-15f0-4c96-86f3-f391d529a791", "node_type": "1", "metadata": {}, "hash": "f972f6c387d124395a2a0fabb2b65591ec6d453ecca3ba9859f831cad21e7b0f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11066, "end_char_idx": 14574, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test the summarize_text function works
chat_test_response = summarize_text("PPO reinforcement learning sequence generation")

```

    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:06<00:00,  1.08s/it]


    Summarizing into overall summary



```python
print(chat_test_response.choices[0].message.content)

```

    Core Argument:
    - The academic paper explores the connection between the transverse field Ising (TFI) model and the ϕ4 model, highlighting the analogy between topological solitary waves in the ϕ4 model and the effect of the transverse field on spin flips in the TFI model.
    - The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical ϕ4 model subjected to periodic perturbations, which are essential in annealing phenomena.
    - The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.
    
    Evidence:
    - The authors analyze the dynamics and relaxation of weakly coupled ϕ4 chains through numerical simulations, observing kink and breather excitations and investigating the structural phase transition associated with the double well potential.
    - The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C⊥) approaches zero, but there is a finite Tc for C⊥>0.
    - The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.
    - The soft-mode frequency (ωs) decreases as temperature approaches Ts, the dynamical crossover temperature.
    - The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.
    - The mean domain size (⟨DS⟩) exhibits an algebraic decay for finite C⊥>0.
    - The probability of larger domain sizes is higher before a kick compared to after a kick for C⊥>0.
    
    Conclusions:
    - The authors suggest further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space.
    - They propose extending the study to different kick profiles, introducing kink defects, and studying weakly-coupled chains in higher dimensions.
    - Recognizing similarities between classical nonlinear equations and quantum linear ones in information processing is important.
    - Future research directions include investigating the dynamics of quantum annealing, measurement and memory in the periodically driven complex Ginzburg-Landau equation, and the behavior of solitons and domain walls in various systems.


## Configure Agent

We'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, functions=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            functions=functions,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


key: _node_content
string_value:
{"id_": "705c20f1-15f0-4c96-86f3-f391d529a791", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "631a23d4-4e44-41d6-aca0-40f0b3af43c8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "202aceb1b7ba433661ff9a1d21ca08b4e64d6ac47dfae01ca3cf4a0c6773cd25", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a49b643b-e2d6-49c1-8a7e-b244ce45fb49", "node_type": "1", "metadata": {}, "hash": "f2c2739b500be4e6dd472eb17df76ed9fb7afeff7e2baf78e6697081e179d747", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13833, "end_char_idx": 17168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Configure Agent

We'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, functions=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            functions=functions,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


```python
class Conversation:
    def __init__(self):
        self.conversation_history = []

    def add_message(self, role, content):
        message = {"role": role, "content": content}
        self.conversation_history.append(message)

    def display_conversation(self, detailed=False):
        role_to_color = {
            "system": "red",
            "user": "green",
            "assistant": "blue",
            "function": "magenta",
        }
        for message in self.conversation_history:
            print(
                colored(
                    f"{message['role']}: {message['content']}\n\n",
                    role_to_color[message["role"]],
                )
            )
```


```python
# Initiate our get_articles and read_article_and_summarize functions
arxiv_functions = [
    {
        "name": "get_articles",
        "description": """Use this function to get academic papers from arXiv to answer user questions.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            User query in JSON. Responses should be summarized and should include the article URL reference
                            """,
                }
            },
            "required": ["query"],
        },
    },
    {
        "name": "read_article_and_summarize",
        "description": """Use this function to read whole papers and provide a summary for users.
        You should NEVER call this function before get_articles has been called in the conversation.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            Description of the article in plain text based on the user's query
                            """,
                }
            },
            "required": ["query"],
        },
    }
]

```


```python
def chat_completion_with_function_execution(messages, functions=[None]):
    """This function makes a ChatCompletion API call with the option of adding functions"""
    response = chat_completion_request(messages, functions)
    full_message = response.choices[0]
    if full_message.finish_reason == "function_call":
        print(f"Function generation requested, calling function")
        return call_arxiv_function(messages, full_message)
    else:
        print(f"Function not required, responding to user")
        return response


key: _node_content
string_value:
{"id_": "a49b643b-e2d6-49c1-8a7e-b244ce45fb49", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "705c20f1-15f0-4c96-86f3-f391d529a791", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d1388e240614fa5fac7593c9eb1972906f54a3cc155e7b603c76ec3cd6ff1bae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87436c88-44b5-41a5-9065-b95cba2a29b6", "node_type": "1", "metadata": {}, "hash": "8d17641a152018161c22c138deb19fbb3810445cb2e00f5e9e44e55ad7bb4c95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15283, "end_char_idx": 19327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate our get_articles and read_article_and_summarize functions
arxiv_functions = [
    {
        "name": "get_articles",
        "description": """Use this function to get academic papers from arXiv to answer user questions.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            User query in JSON. Responses should be summarized and should include the article URL reference
                            """,
                }
            },
            "required": ["query"],
        },
    },
    {
        "name": "read_article_and_summarize",
        "description": """Use this function to read whole papers and provide a summary for users.
        You should NEVER call this function before get_articles has been called in the conversation.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            Description of the article in plain text based on the user's query
                            """,
                }
            },
            "required": ["query"],
        },
    }
]

```


```python
def chat_completion_with_function_execution(messages, functions=[None]):
    """This function makes a ChatCompletion API call with the option of adding functions"""
    response = chat_completion_request(messages, functions)
    full_message = response.choices[0]
    if full_message.finish_reason == "function_call":
        print(f"Function generation requested, calling function")
        return call_arxiv_function(messages, full_message)
    else:
        print(f"Function not required, responding to user")
        return response


def call_arxiv_function(messages, full_message):
    """Function calling function which executes function calls when the model believes it is necessary.
    Currently extended by adding clauses to this if statement."""

    if full_message.message.function_call.name == "get_articles":
        try:
            parsed_output = json.loads(
                full_message.message.function_call.arguments
            )
            print("Getting search results")
            results = get_articles(parsed_output["query"])
        except Exception as e:
            print(parsed_output)
            print(f"Function execution failed")
            print(f"Error message: {e}")
        messages.append(
            {
                "role": "function",
                "name": full_message.message.function_call.name,
                "content": str(results),
            }
        )
        try:
            print("Got search results, summarizing content")
            response = chat_completion_request(messages)
            return response
        except Exception as e:
            print(type(e))
            raise Exception("Function chat request failed")

    elif (
        full_message.message.function_call.name == "read_article_and_summarize"
    ):
        parsed_output = json.loads(
            full_message.message.function_call.arguments
        )
        print("Finding and reading paper")
        summary = summarize_text(parsed_output["query"])
        return summary

    else:
        raise Exception("Function does not exist and cannot be called")

```

## arXiv conversation

Let's put this all together by testing our functions out in conversation.


```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


key: _node_content
string_value:
{"id_": "87436c88-44b5-41a5-9065-b95cba2a29b6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a49b643b-e2d6-49c1-8a7e-b244ce45fb49", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "880985dc24aac144d7436f46d8a05fb03dbd7f87f8391ed9d30fe2d22b71e41c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0671465c-7c16-48da-a8f2-3c5ceb188b34", "node_type": "1", "metadata": {}, "hash": "080f1c8dcfd97e8e4912db3e5715f056283b13c875b328da309ff87e2424e2cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17171, "end_char_idx": 19868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def call_arxiv_function(messages, full_message):
    """Function calling function which executes function calls when the model believes it is necessary.
    Currently extended by adding clauses to this if statement."""

    if full_message.message.function_call.name == "get_articles":
        try:
            parsed_output = json.loads(
                full_message.message.function_call.arguments
            )
            print("Getting search results")
            results = get_articles(parsed_output["query"])
        except Exception as e:
            print(parsed_output)
            print(f"Function execution failed")
            print(f"Error message: {e}")
        messages.append(
            {
                "role": "function",
                "name": full_message.message.function_call.name,
                "content": str(results),
            }
        )
        try:
            print("Got search results, summarizing content")
            response = chat_completion_request(messages)
            return response
        except Exception as e:
            print(type(e))
            raise Exception("Function chat request failed")

    elif (
        full_message.message.function_call.name == "read_article_and_summarize"
    ):
        parsed_output = json.loads(
            full_message.message.function_call.arguments
        )
        print("Finding and reading paper")
        summary = summarize_text(parsed_output["query"])
        return summary

    else:
        raise Exception("Function does not exist and cannot be called")

```

## arXiv conversation

Let's put this all together by testing our functions out in conversation.


```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


```python
# Add a user message
paper_conversation.add_message("user", "Hi, how does PPO reinforcement learning work?")
chat_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
assistant_message = chat_response.choices[0].message.content
paper_conversation.add_message("assistant", assistant_message)
display(Markdown(assistant_message))

```

    Function generation requested, calling function
    Getting search results
    Got search results, summarizing content


key: _node_content
string_value:
{"id_": "0671465c-7c16-48da-a8f2-3c5ceb188b34", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87436c88-44b5-41a5-9065-b95cba2a29b6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a1e746dd834dac2580d078e7ca05bb48e7acd2d9cfdb159c131ea106b9e45d11", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a6ae21f-bdb9-4a90-94d6-6d73f6b5ec0d", "node_type": "1", "metadata": {}, "hash": "925f77bc44c7ace54858e6dfd2cb44cae9d97e6e9b6542607e6a17b25effd08c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18836, "end_char_idx": 22344, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


```python
# Add a user message
paper_conversation.add_message("user", "Hi, how does PPO reinforcement learning work?")
chat_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
assistant_message = chat_response.choices[0].message.content
paper_conversation.add_message("assistant", assistant_message)
display(Markdown(assistant_message))

```

    Function generation requested, calling function
    Getting search results
    Got search results, summarizing content



PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that aims to find the optimal policy for an agent by optimizing the policy parameters in an iterative manner. Here are a few papers that discuss PPO in more detail:

1. Title: "Proximal Policy Optimization Algorithms"
   Article URL: [arxiv.org/abs/1707.06347v2](http://arxiv.org/abs/1707.06347v2)
   Summary: This paper introduces two algorithms, PPO (Proximal Policy Optimization) and TRPO (Trust Region Policy Optimization), that address the issue of sample efficiency and stability in reinforcement learning. PPO uses a surrogate objective function that makes smaller updates to the policy parameters, resulting in more stable and efficient learning.

2. Title: "Emergence of Locomotion Behaviours in Rich Environments with PPO"
   Article URL: [arxiv.org/abs/1707.02286v3](http://arxiv.org/abs/1707.02286v3)
   Summary: This paper explores the use of PPO in training agents to learn locomotion behaviors in complex and dynamic environments. The authors demonstrate the effectiveness of PPO in learning a variety of locomotion skills, such as walking, jumping, and climbing.

3. Title: "Proximal Policy Optimization for Multi-Agent Systems"
   Article URL: [arxiv.org/abs/2006.14171v2](http://arxiv.org/abs/2006.14171v2)
   Summary: This paper extends PPO to the domain of multi-agent systems, where multiple agents interact and learn together. The authors propose a decentralized version of PPO that allows each agent to update its policy independently based on its local observations, resulting in more scalable and efficient learning in multi-agent environments.

These papers provide detailed explanations of the PPO algorithm, its advantages, and its applications in different scenarios. Reading them can give you a deeper understanding of how PPO reinforcement learning works.



```python
# Add another user message to induce our system to use the second tool
paper_conversation.add_message(
    "user",
    "Can you read the PPO sequence generation paper for me and give me a summary",
)
updated_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
display(Markdown(updated_response.choices[0].message.content))

```

    Function generation requested, calling function
    Finding and reading paper
    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:07<00:00,  1.19s/it]


key: _node_content
string_value:
{"id_": "1a6ae21f-bdb9-4a90-94d6-6d73f6b5ec0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0671465c-7c16-48da-a8f2-3c5ceb188b34", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "467260934d65843d64863bfe80d7618ecb2ddeeac6279b8a65e899d4a7014647", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21735, "end_char_idx": 24713, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Add another user message to induce our system to use the second tool
paper_conversation.add_message(
    "user",
    "Can you read the PPO sequence generation paper for me and give me a summary",
)
updated_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
display(Markdown(updated_response.choices[0].message.content))

```

    Function generation requested, calling function
    Finding and reading paper
    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:07<00:00,  1.19s/it]


    Summarizing into overall summary



Core Argument:
- The academic paper explores the connection between the transverse field Ising (TFI) model and the ϕ4 model, highlighting the analogy between the coupling of topological solitary waves in the ϕ4 model and the effect of the transverse field on spin flips in the TFI model.
- The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical ϕ4 model subjected to periodic perturbations, which are essential in annealing phenomena.
- The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.

Evidence:
- The authors analyze the dynamics and relaxation of weakly coupled ϕ4 chains through numerical simulations, studying the behavior of kink and breather excitations and the structural phase transition associated with the double well potential.
- The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C⊥) approaches zero, but there is a finite Tc for C⊥>0.
- The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.
- The soft-mode frequency (ωs) decreases as temperature approaches Ts, the dynamical crossover temperature.
- The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.
- The mean domain size (⟨DS⟩) exhibits an algebraic decay for finite C⊥>0.
- The probability of larger domain sizes is higher before a kick compared to after a kick for C⊥>0.

Conclusions:
- The study of weakly-coupled classical ϕ4 chains provides insights into quantum annealing architectures and the role of topological excitations in these systems.
- The equilibration of the system is faster for higher kick strengths, and the mean domain size increases with higher final temperatures.
- Further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space is suggested.
- The paper highlights the importance of recognizing similarities between classical nonlinear equations and quantum linear ones in information processing and suggests future research directions in this area.


key: _node_content
string_value:
{"id_": "5072734c-13c4-4ca6-9e19-ccd77a28dfaf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54a9a06e-97ba-44f9-a854-5d10a9aed824", "node_type": "1", "metadata": {}, "hash": "49ee01e928a34ae21f1de9c7e8e3725cbaab931199fb73255f1b385b51faa3eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Synthetic Data generation (Part 1)


Synthetic data generation using large language models (LLMs) offers a powerful solution to a commonly faced problem: the availability of high-quality, diverse, and privacy-compliant data. This could be used in a number of scenarios such as training a data science  machine learning model (SVMs, decision trees, KNN's), finetuning a different GPT model on the data, as a solution to the coldstart problem, helping build compelling demos/apps with realistic data, scenario testing etc.

There are a number of key drivers which may see you wanting to leverage synthetic data. 
1. Human data may have privacy restrictions and/or identifiable data within it which we do not want to be used. 
2. Synthetic data can be much more structured and therefore easier to manipulate than real data. 
3. In domains where data is sparse or data of certain categories is sparse we may want to augment the data. 
4. When dealing with imbalanced datasets or datasets which lack diversity, we may want to create data to improve the richness of our datasets.

Unlike traditional data augmentation or manual data creation methods, using LLMs allows for the generation of rich, nuanced, and contextually relevant datasets that can significantly enhance it's usefulness to enterprises and developers.

We split this tutorial into 2 parts. In this cookbook, we will have the following agenda:
1. CSV with a structured prompt
2. CSV with a Python program
3. Multitable CSV with a python program
4. Simply creating textual data
5. Dealing with imbalanced or non-diverse textual data
while in part 2, we will look at prompting strategies for getting better textual data.

The last two in particular are useful for creating synthetic data to finetune another GPT model. For example using higher quality data produced by `gpt-4o` to finetune the cheaper and quicker `gpt-3.5-turbo` for improved performance while reducing costs.


### Getting setup


```python
%pip install openai
%pip install pandas
%pip install scikit-learn
%pip install matplotlib
```


```python
from openai import OpenAI
import os
import re
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import json
import matplotlib

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. CSV with a structure prompt
Here we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance).


key: _node_content
string_value:
{"id_": "54a9a06e-97ba-44f9-a854-5d10a9aed824", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5072734c-13c4-4ca6-9e19-ccd77a28dfaf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1a10a5708848ffdc9b96923f94f67c7df9cf591a860b78e04dee0829e66042ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e26ce35e-be5f-4c81-864f-c20b71ac5f71", "node_type": "1", "metadata": {}, "hash": "2c7a757767a39d3c8af34856581764743b042af8eda1205092cc674d8f566f9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1939, "end_char_idx": 4373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Getting setup


```python
%pip install openai
%pip install pandas
%pip install scikit-learn
%pip install matplotlib
```


```python
from openai import OpenAI
import os
import re
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import json
import matplotlib

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. CSV with a structure prompt
Here we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance).


```python
datagen_model = "gpt-4o-mini"
question = """
Create a CSV file with 10 rows of housing data.
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense). Also only respond with the CSV.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    ```csv
    id,house_size_m2,house_price,location,number_of_bedrooms
    1,50,150000,Suburban,2
    2,75,250000,City Center,3
    3,100,350000,Suburban,4
    4,120,450000,Suburban,4
    5,80,300000,City Center,3
    6,90,400000,City Center,3
    7,150,600000,Premium Area,5
    8,200,750000,Premium Area,5
    9,55,180000,Suburban,2
    10,300,950000,Premium Area,6
    ```


### 2. CSV with a Python program
The issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.

This would then let us edit the python program as we desire while giving us a good basis to start from.


key: _node_content
string_value:
{"id_": "e26ce35e-be5f-4c81-864f-c20b71ac5f71", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54a9a06e-97ba-44f9-a854-5d10a9aed824", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9813953a1dd63f836586d838f9e966e66d49c5832896e589a9f66292182b1ff8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e4b3853-1cdb-495d-99e8-85d626030bde", "node_type": "1", "metadata": {}, "hash": "aa076a8765b2bab84cd13af17a8842a34f01bbde49883af51ed15f092f67a5f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3878, "end_char_idx": 7670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 2. CSV with a Python program
The issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.

This would then let us edit the python program as we desire while giving us a good basis to start from.



```python
question = """
Create a Python program to generate 100 rows of housing data.
I want you to at the end of it output a pandas dataframe with 100 rows of data.
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates synthetic housing data according to your specifications. We will create a pandas DataFrame with the defined fields and characteristics.
    
    ```python
    import pandas as pd
    import random
    
    def generate_housing_data(num_rows):
        data = []
        
        locations = [
            ('City Center', 10000, 150),  # (location name, base price per m², base size)
            ('Suburban Area', 8000, 100),
            ('Country Side', 5000, 80),
            ('Coastal Region', 12000, 110),
            ('Urban Neighborhood', 9000, 130)
        ]
        
        for i in range(1, num_rows + 1):
            # Randomly pick a location
            location, base_price_per_m2, base_size = random.choice(locations)
            
            # Generate number of bedrooms (1 to 5)
            number_of_bedrooms = random.randint(1, 5)
            
            # Calculate house size based on the number of bedrooms
            house_size = base_size + (10 * number_of_bedrooms) + random.randint(-5, 15)  # Adding some noise
            
            # Calculate house price based on house size and location
            house_price = base_price_per_m2 * house_size + random.randint(-5000, 10000)  # Adding some noise
    
            # Append the generated data to the list
            data.append({
                'id': i,
                'house_size_m2': house_size,
                'house_price': house_price,
                'location': location,
                'number_of_bedrooms': number_of_bedrooms
            })
    
        # Create a pandas DataFrame
        df = pd.DataFrame(data)
        return df
    
    # Generate 100 rows of housing data
    housing_data_df = generate_housing_data(100)
    
    # Show the result
    print(housing_data_df)
    ```
    
    ### Explanation:
    - The `generate_housing_data` function creates synthetic housing data for a specified number of rows (`num_rows`).
    - We define different locations with corresponding base prices per square meter and average house sizes.
    - For each house, we randomly select a location, number of bedrooms, and calculate house size and price to ensure a sensible correlation between the values.
    - Finally, we create a pandas DataFrame from the generated data and return it.
    
    You can run this program in your Python environment, and it will output a DataFrame containing 100 rows of synthetic housing data.


key: _node_content
string_value:
{"id_": "9e4b3853-1cdb-495d-99e8-85d626030bde", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e26ce35e-be5f-4c81-864f-c20b71ac5f71", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f0a120eb82303e230e9335b1a2f2c04a99eb3665ade9a4402d3c92d375628e59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4a26182-76ba-4fbc-85e3-d49b0bd57869", "node_type": "1", "metadata": {}, "hash": "16ea581358e7938df7828f6d4365438374b63286c05a0564675d4dd9151b90b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7673, "end_char_idx": 10470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We need to make sure to parse the output of this appropriately as often there may be surrounding text to the python code. We can also explicitly ask it to state all assumptions it made about the data it's generating, however in this circumstance it told us that automatically.

### 3. Multitable CSV with a python program
For more complex relationships however we need to make sure to specify a few more characteristics. 

To create multiple different datasets which relate to each other (for example housing, location, house type), as before we would need to specify the format, schema and useful information. However, the useful information required to get good performance is higher now. It's case-specific but a good amount of things to describe would be how the datasets relate to each other, addressing the size of the datasets in relation to one another, making sure foreign and primary keys are made appropriately and ideally using previously generated datasets to populate new ones so the actual data values match where necessary.


```python
question = """
Create a Python program to generate 3 different pandas dataframes.

1. Housing data
I want 100 rows. Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms
 - house type
 + any relevant foreign keys

2. Location
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - country
 - city
 - population
 - area (m^2)
 + any relevant foreign keys

 3. House types
 - id (incrementing integer starting at 1)
 - house type
 - average house type price
 - number of houses
 + any relevant foreign keys

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
Make sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.
Make sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.
You can use the previously generated dataframe to generate the next dataframe.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them.


key: _node_content
string_value:
{"id_": "c4a26182-76ba-4fbc-85e3-d49b0bd57869", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e4b3853-1cdb-495d-99e8-85d626030bde", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1c7026332cd9b58ad1c5082ec71e7aebee03f7965f95ced5517bb41dfee624f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da82c1cc-0d6b-4efe-bd0b-7a55417f4fca", "node_type": "1", "metadata": {}, "hash": "c85a96c211ebb27c0096724b202f1b0077c6a0a49b17f0faf10be94b59e32abe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8715, "end_char_idx": 13481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
question = """
Create a Python program to generate 3 different pandas dataframes.

1. Housing data
I want 100 rows. Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms
 - house type
 + any relevant foreign keys

2. Location
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - country
 - city
 - population
 - area (m^2)
 + any relevant foreign keys

 3. House types
 - id (incrementing integer starting at 1)
 - house type
 - average house type price
 - number of houses
 + any relevant foreign keys

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
Make sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.
Make sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.
You can use the previously generated dataframe to generate the next dataframe.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them.
    
    ```python
    import pandas as pd
    import numpy as np
    
    # Set random seed for reproducibility
    np.random.seed(0)
    
    # Function to generate location DataFrame
    def generate_location_data(num_locations):
        locations = {
            "id": range(1, num_locations + 1),
            "country": np.random.choice(['USA', 'Canada', 'UK'], num_locations),
            "city": np.random.choice(['New York', 'Toronto', 'London', 'Vancouver', 'Manchester'], num_locations),
            "population": np.random.randint(50000, 1000000, num_locations),
            "area": np.random.randint(10000, 500000, num_locations)
        }
        return pd.DataFrame(locations)
    
    # Function to generate house types DataFrame
    def generate_house_type_data(num_house_types):
        house_types = {
            "id": range(1, num_house_types + 1),
            "house_type": np.random.choice(['Detached', 'Semi-Detached', 'Terraced', 'Flat'], num_house_types),
            "average_house_type_price": np.random.randint(100000, 1000000, num_house_types),
            "number_of_houses": np.random.randint(10, 1000, num_house_types)
        }
        return pd.DataFrame(house_types)
    
    # Function to generate housing data DataFrame
    def generate_housing_data(num_houses, location_df, house_type_df):
        house_sizes = np.random.randint(50, 300, num_houses)  # size in m^2
        location_ids = np.random.choice(location_df['id'], num_houses)
        house_type_ids = np.random.choice(house_type_df['id'], num_houses)
        
        # Generate prices based on size, location, and house type
        house_prices = (house_sizes * np.random.randint(2000, 5000, num_houses) // 10) + \
                       (location_ids * 1000) + \
                       (house_type_df.loc[house_type_ids - 1, 'average_house_type_price'].values // 4)
        
        housing_data = {
            "id": range(1, num_houses + 1),
            "house_size": house_sizes,
            "house_price": house_prices,
            "location_id": location_ids,
            "bedrooms": np.random.randint(1, 6, num_houses),
            "house_type_id": house_type_ids
        }
        
        return pd.DataFrame(housing_data)
    
    # Generate DataFrames
    num_locations = 10
    num_house_types = 4
    num_houses = 100
    
    location_df = generate_location_data(num_locations)
    house_type_df = generate_house_type_data(num_house_types)
    housing_df = generate_housing_data(num_houses, location_df, house_type_df)
    
    # Display the generated DataFrames
    print("Location DataFrame:")
    print(location_df.head(), "\n")
    
    print("House Types DataFrame:")
    print(house_type_df.head(), "\n")
    
    print("Housing DataFrame:")
    print(housing_df.head(), "\n")
    
    # Printing the DataFrame shapes
    print(f"Shapes: \nLocation: {location_df.shape}, House Types: {house_type_df.shape}, Housing: {housing_df.shape}")
    ```
    
    ### Explanation of the Code:
    1.


key: _node_content
string_value:
{"id_": "da82c1cc-0d6b-4efe-bd0b-7a55417f4fca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4a26182-76ba-4fbc-85e3-d49b0bd57869", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2edf50dcc5d1d1108d00fd221879b17f02ee59fa73376a29682db268cf7f9fd1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "972d5409-cc6d-4212-8a8b-d44d566828b9", "node_type": "1", "metadata": {}, "hash": "f3d0f24dda79e5e2b243302f6bffa8e321ed8eaa8a8816f7a15c475f6e7d2e43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13482, "end_char_idx": 15207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Location DataFrame:** 
       - Generates random locations with attributes such as country, city, population, and area.
      
    2. **House Types DataFrame:** 
       - Generates different types of houses along with average prices and quantity available.
      
    3. **Housing DataFrame:** 
       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.
    
    ### Output:
    The three DataFrames generated will logically relate to one another with consistent data types and primary–foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.


### 4. Simply creating textual data
Here we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.

The example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. 

Specifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break.


key: _node_content
string_value:
{"id_": "972d5409-cc6d-4212-8a8b-d44d566828b9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da82c1cc-0d6b-4efe-bd0b-7a55417f4fca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1c3d80a2ccf971177b75ccc4402e4e93937b033ed70d058bdcff4d3d1c2e3a02", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e94d1f21-c796-4e00-a4ab-a8792f0fd210", "node_type": "1", "metadata": {}, "hash": "107dcd1b16bd56ac232db0ef144fbcbcd62ee166a03bdbe078efb573ffa19887", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13482, "end_char_idx": 17323, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Location DataFrame:** 
       - Generates random locations with attributes such as country, city, population, and area.
      
    2. **House Types DataFrame:** 
       - Generates different types of houses along with average prices and quantity available.
      
    3. **Housing DataFrame:** 
       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.
    
    ### Output:
    The three DataFrames generated will logically relate to one another with consistent data types and primary–foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.


### 4. Simply creating textual data
Here we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.

The example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. 

Specifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. The usecase is a retailer generating a description for a product from a product catalogue. I want the input to be product name and category (to which the product belongs to) and output to be description.
  The format should be of the form:
  1.
  Input: product_name, category
  Output: description
  2.
  Input: product_name, category
  Output: description

  Do not add any extra characters around that formatting as it will make the output parsing break.
  Create as many training pairs as possible.
  """

  response = client.chat.completions.create(
    model=datagen_model,
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string[:1000]) #displaying truncated response

```

    1.
    Input: Wireless Bluetooth Headphones, Electronics
    Output: Immerse yourself in high-quality sound with these Wireless Bluetooth Headphones, featuring active noise cancellation and a comfortable over-ear design for extended listening sessions.
    
    2.
    Input: Organic Green Tea, Beverages
    Output: Enjoy a refreshing cup of Organic Green Tea, sourced from the finest leaves, packed with antioxidants, and perfect for a healthy, invigorating boost anytime.
    
    3.
    Input: Stainless Steel Kitchen Knife, Kitchenware
    Output: Cut with precision and ease using this Stainless Steel Kitchen Knife, designed with an ergonomic handle and a sharp blade for all your culinary tasks.
    
    4.
    Input: Hiking Backpack, Outdoor Gear
    Output: Explore the great outdoors with this durable Hiking Backpack, featuring multiple compartments for optimal organization and a breathable design for ultimate comfort on long treks.
    
    5.
    Input: Air Fryer, Kitchen Appliances
    Output: Cook your favorite meals with less oil using this Air Fryer


key: _node_content
string_value:
{"id_": "e94d1f21-c796-4e00-a4ab-a8792f0fd210", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "972d5409-cc6d-4212-8a8b-d44d566828b9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c0ad067b3e86db1111a51a4977c3dd87c6c7a28c3f43676415aeb665bde52429", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52689d2a-c4b8-439a-9ff6-7905d23b8e4c", "node_type": "1", "metadata": {}, "hash": "bb028639b0decedb89bbdee3201b4e6adcdf2a7455c60e40e2ed1b2b73a04829", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17326, "end_char_idx": 20387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note: the above output is truncated. And now we can parse it as below to get a list of products, categories and their descriptions. For example, let's take a look at the products it's generated.


```python
#regex to parse data
pattern = re.compile(r'Input:\s*(.+?),\s*(.+?)\nOutput:\s*(.+?)(?=\n\n|\Z)', re.DOTALL)
matches = pattern.findall(output_string)
products = []
categories = []
descriptions = []

for match in matches:
    product, category, description = match
    products.append(product.strip())
    categories.append(category.strip())
    descriptions.append(description.strip())
products
```




    ['Wireless Bluetooth Headphones',
     'Organic Green Tea',
     'Stainless Steel Kitchen Knife',
     'Hiking Backpack',
     'Air Fryer',
     "Kids' Educational Tablet",
     'Bluetooth Speaker',
     'Yoga Mat',
     'Memory Foam Mattress',
     'Smartwatch',
     'Leather Wallet',
     'Portable Phone Charger',
     'Non-Stick Cookware Set',
     'Pet Dog Bed',
     'Fitness Tracker',
     'Wireless Earbuds',
     'Organic Green Tea',
     'Reusable Water Bottle',
     'Yoga Mat',
     'Leather Wallet',
     'Air Fryer',
     'Gaming Mouse',
     'Crochet Kit',
     'Hiking Boots',
     'Scented Candles',
     'Bluetooth Speaker',
     'Stainless Steel Cookware Set',
     'Fitness Tracker',
     'Decorative Throw Pillows',
     'Eco-Friendly Cleaning Supplies',
     'Wireless Noise Cancelling Headphones',
     'Organic Green Tea',
     'Adjustable Yoga Mat',
     'Bluetooth Smart Scale',
     'Stainless Steel Water Bottle',
     'Soft Cotton Bedding Set',
     'Multi-Functional Kitchen Blender',
     'Eco-Friendly Reusable Bags',
     'Portable Phone Charger',
     'Classic Leather Wallet',
     'Suede Chelsea Boots',
     'Non-Stick Cookware Set',
     'Pet-Friendly Indoor Plants',
     'High-Protein Snack Bars',
     'LED Desk Lamp with USB Port']




### 5. Dealing with imbalanced or non-diverse textual data
Some of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).


To increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. 

We can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.

For demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas.


key: _node_content
string_value:
{"id_": "52689d2a-c4b8-439a-9ff6-7905d23b8e4c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e94d1f21-c796-4e00-a4ab-a8792f0fd210", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ba353dc294267170046129ee6184fe381d66d1355e9379280da9fb5564241a2f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb8101d3-6bb5-4f7e-9467-75b699c088aa", "node_type": "1", "metadata": {}, "hash": "9f643c5e3db59aa48aa3ce1771d1fbf72f2826afcfb6188005c1da8be7558526", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19219, "end_char_idx": 22835, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 5. Dealing with imbalanced or non-diverse textual data
Some of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).


To increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. 

We can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.

For demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. I want the input to be product name and category and output to be description. the category should be things like: mobile phones, shoes, headphones, laptop, electronic toothbrush, etc. and also more importantly the categories should come under 4 main topics: vehicle, clothing, toiletries, food)
  After the number of each example also state the topic area. The format should be of the form:
  1. topic_area
  Input: product_name, category
  Output: description

  Do not add any extra characters around that formatting as it will make the output parsing break.

  Here are some helpful examples so you get the style of output correct.

  1) clothing
  Input: "Shoe Name, Shoes"
  Output: "Experience unparalleled comfort. These shoes feature a blend of modern style and the traditional superior cushioning, perfect for those always on the move."
  """

  response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string[:1000]) #displaying truncated response
```

    1. vehicle  
    Input: "Tesla Model 3, Electric Car"  
    Output: "The Tesla Model 3 is a revolutionary electric car with impressive range and cutting-edge technology, designed to provide an exhilarating driving experience while minimizing environmental impact."
    
    2. clothing  
    Input: "Nike Air Max, Shoes"  
    Output: "Elevate your sneaker game with Nike Air Max. Combining iconic style with superior comfort and support, these shoes are perfect for both workouts and casual outings."
    
    3. toiletries  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "Achieve a superior clean with the Oral-B Pro 1000. This electronic toothbrush features 3D cleaning action that pulsates and oscillates to remove more plaque than a regular manual toothbrush."
    
    4. food  
    Input: "Chobani Greek Yogurt, Yogurt"  
    Output: "Indulge in a nutritious snack with Chobani Greek Yogurt. Packed with protein and delicious flavors, it’s the perfect choice for a healthy breakfast or a satisfying treat anytime."
    
    5. vehicle


key: _node_content
string_value:
{"id_": "bb8101d3-6bb5-4f7e-9467-75b699c088aa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52689d2a-c4b8-439a-9ff6-7905d23b8e4c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "89ba8f3ef7b297d2c9109e3f87087a07bbf2e0753e28d93d2ae5cbaf01315863", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "812a6b7e-e434-4724-bf78-021fc80f47f7", "node_type": "1", "metadata": {}, "hash": "ee7ce49eda7a801dcd0d4ceb6c7d054839655873f96cd66edbe77632f040cba8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22845, "end_char_idx": 25758, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note: The above output is truncated. In the example above, we would explicitly include the topic area as part of the response per example as it helps condition the proceeding output and tends to give better performance. We can also give it an actual example of what the output should look like so it gets the right idea of style of output but also to help enforce structure.


```python
pattern = re.compile(r'(\d+)\.\s*(\w+)\s*Input:\s*"(.+?),\s*(.+?)"\s*Output:\s*"(.*?)"', re.DOTALL)
matches = pattern.findall(output_string)

topics = []
products = []
categories = []
descriptions = []

for match in matches:
    number, topic, product, category, description = match
    topics.append(topic)
    products.append(product)
    categories.append(category)
    descriptions.append(description)

```


```python
products
```




    ['Tesla Model 3',
     'Nike Air Max',
     'Oral-B Pro 1000',
     'Chobani Greek Yogurt',
     'Ford F-150',
     "Levi's 511",
     'Philips Sonicare',
     'Quaker Oatmeal',
     'Toyota Camry',
     'Adidas Ultraboost',
     'Toyota Camry',
     'Nike Air Max',
     'Colgate Electric Toothbrush',
     'Blue Diamond Almonds',
     'Harley Davidson Fat Boy',
     'Adidas UltraBoost',
     "Dove Men's Body Wash",
     'Quaker Oats',
     'Ford F-150',
     "Levi's 501 Jeans",
     'Tesla Model 3',
     'Nike Air Max',
     'Oral-B Pro 1000',
     'Organic Almond Butter',
     'Yamaha YZF-R3',
     'Adidas Ultraboost',
     'Philips Sonicare',
     'Organic Quinoa']



We will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.

We know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.

In the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).

First let's store our data into a pandas dataframe for ease of analysis




```python
data = {
    'Product': products,
    'Category': categories,
    'Description': descriptions
}

df = pd.DataFrame(data)
```

Next let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar.


key: _node_content
string_value:
{"id_": "812a6b7e-e434-4724-bf78-021fc80f47f7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb8101d3-6bb5-4f7e-9467-75b699c088aa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "85ebcc0ee837570505773a92f0261ee9ce777aae84b8c566365229448b9a8cbe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49315982-5eb9-40eb-8db8-1b1ea6bab948", "node_type": "1", "metadata": {}, "hash": "f36525c142765e26c98342b4f3d0733fbfca7f344f4e5a7a9430c42f0f7a03d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24355, "end_char_idx": 27149, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.

We know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.

In the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).

First let's store our data into a pandas dataframe for ease of analysis




```python
data = {
    'Product': products,
    'Category': categories,
    'Description': descriptions
}

df = pd.DataFrame(data)
```

Next let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar.


```python
def get_embedding(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")

    response = client.embeddings.create(input=[text], model=model)

    return response.data[0].embedding

embedding_model = "text-embedding-3-small"
df["embedding"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))

# Ensure there are embeddings to concatenate
if len(df.embedding.values) > 0:
    matrix = np.vstack(df.embedding.values)
else:
    matrix = np.array([])  # Handle the case where there are no embeddings

```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max.


key: _node_content
string_value:
{"id_": "49315982-5eb9-40eb-8db8-1b1ea6bab948", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "812a6b7e-e434-4724-bf78-021fc80f47f7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c5d4f529248801dd18d07ff88c00bb737223859be44d55c9b7119cef1f902b10", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6429b49b-d791-4506-80a6-440a6cb5d114", "node_type": "1", "metadata": {}, "hash": "e617f7016719dc7cbcd888bfc804faf08fc64faedc223d08d90222ce9daa5cd0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25761, "end_char_idx": 28036, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_embedding(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")

    response = client.embeddings.create(input=[text], model=model)

    return response.data[0].embedding

embedding_model = "text-embedding-3-small"
df["embedding"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))

# Ensure there are embeddings to concatenate
if len(df.embedding.values) > 0:
    matrix = np.vstack(df.embedding.values)
else:
    matrix = np.array([])  # Handle the case where there are no embeddings

```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max. C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans.


key: _node_content
string_value:
{"id_": "6429b49b-d791-4506-80a6-440a6cb5d114", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49315982-5eb9-40eb-8db8-1b1ea6bab948", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ef822fdc2c9312685937245c2e3277d173c47e8ea813c8058a04137d3e4d4c09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6cab60bd-c4bd-42c6-886c-9ec73524eefa", "node_type": "1", "metadata": {}, "hash": "c5bd4840b32918054101fb5947705345a4143cd4a4b884578091fc676a7af83c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27150, "end_char_idx": 28498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans. Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal.


key: _node_content
string_value:
{"id_": "6cab60bd-c4bd-42c6-886c-9ec73524eefa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6429b49b-d791-4506-80a6-440a6cb5d114", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d956d8dab1eb90f2228c51e7ac20b9d9c8e9141153809cf20b964d250a5ee03d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fce0d66-d71a-45d0-afb0-8ed05824735b", "node_type": "1", "metadata": {}, "hash": "b0794f7573912c1050eecc2938c42d5d4eb5e05ed924fe59629c008d2b697f6f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28037, "end_char_idx": 29840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal. This...</td>
      <td>[-0.006529285106807947, 0.007865572348237038, ...</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Toyota Camry</td>
      <td>Sedan</td>
      <td>The Toyota Camry stands out in the sedan categ...</td>
      <td>[-0.02088991366326809, -0.006191295105963945, ...</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Adidas Ultraboost</td>
      <td>Running Shoes</td>
      <td>Run like never before in the Adidas Ultraboost...</td>
      <td>[0.02679188922047615, 0.014639599248766899, 8....</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Toyota Camry</td>
      <td>Car</td>
      <td>The Toyota Camry is a reliable midsize sedan k...</td>
      <td>[0.008056452497839928, -0.007912316359579563, ...</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your sneaker game with the Nike Air Ma...</td>
      <td>[0.03943241760134697, 0.02208484522998333, -0....</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Colgate Electric Toothbrush</td>
      <td>Electronic Toothbrush</td>
      <td>Transform your oral hygiene routine with the C...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>13</th>
      <td>Blue Diamond Almonds</td>
      <td>Nuts</td>
      <td>Snack healthy with Blue Diamond Almonds.


key: _node_content
string_value:
{"id_": "2fce0d66-d71a-45d0-afb0-8ed05824735b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6cab60bd-c4bd-42c6-886c-9ec73524eefa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "668a01e1d28b2a7257f96188c448782c1d099bdc26b5bdf5efbddba90e9f0597", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "affff5c6-4a1f-49d4-953b-67dd05ec25e0", "node_type": "1", "metadata": {}, "hash": "ef8dd0a14988b2c3378f6cbd7481810d95c39add1aa7c595908eac9b08eaf7a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29841, "end_char_idx": 31613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These...</td>
      <td>[-0.013289917260408401, 0.036334190517663956, ...</td>
    </tr>
    <tr>
      <th>14</th>
      <td>Harley Davidson Fat Boy</td>
      <td>Motorcycle</td>
      <td>Experience the thrill of the open road with th...</td>
      <td>[0.012365399859845638, 0.03552943095564842, -0...</td>
    </tr>
    <tr>
      <th>15</th>
      <td>Adidas UltraBoost</td>
      <td>Sneakers</td>
      <td>Enjoy a perfect blend of comfort and performan...</td>
      <td>[0.013107392005622387, 0.02963760495185852, -0...</td>
    </tr>
    <tr>
      <th>16</th>
      <td>Dove Men's Body Wash</td>
      <td>Body Wash</td>
      <td>Refresh and hydrate your skin with Dove Men's ...</td>
      <td>[0.03760576993227005, -0.008475445210933685, -...</td>
    </tr>
    <tr>
      <th>17</th>
      <td>Quaker Oats</td>
      <td>Oats</td>
      <td>Start your day right with Quaker Oats. Packed ...</td>
      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Ford F-150</td>
      <td>Truck</td>
      <td>The Ford F-150 is a durable and dependable tru...</td>
      <td>[0.023461222648620605, -0.026651185005903244, ...</td>
    </tr>
    <tr>
      <th>19</th>
      <td>Levi's 501 Jeans</td>
      <td>Jeans</td>
      <td>Discover the timeless style of Levi's 501 Jean...</td>
      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>
    </tr>
    <tr>
      <th>20</th>
      <td>Tesla Model 3</td>
      <td>Mobile Phones</td>
      <td>Explore the future of driving with the Tesla M...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
    </tr>
    <tr>
      <th>21</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your game with the Nike Air Max.


key: _node_content
string_value:
{"id_": "affff5c6-4a1f-49d4-953b-67dd05ec25e0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fce0d66-d71a-45d0-afb0-8ed05824735b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0c4ef7676d35ad51d9ff9881c2d0825c195fd8915df7656ce6fd732ed22c3da6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42a1def6-820a-4294-ad5e-9b733a261a0f", "node_type": "1", "metadata": {}, "hash": "7a775869be6e9f7ffad94df167ce3a3f1d7bec72f2488b49f32026a44645a77b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30737, "end_char_idx": 33087, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Packed ...</td>
      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Ford F-150</td>
      <td>Truck</td>
      <td>The Ford F-150 is a durable and dependable tru...</td>
      <td>[0.023461222648620605, -0.026651185005903244, ...</td>
    </tr>
    <tr>
      <th>19</th>
      <td>Levi's 501 Jeans</td>
      <td>Jeans</td>
      <td>Discover the timeless style of Levi's 501 Jean...</td>
      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>
    </tr>
    <tr>
      <th>20</th>
      <td>Tesla Model 3</td>
      <td>Mobile Phones</td>
      <td>Explore the future of driving with the Tesla M...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
    </tr>
    <tr>
      <th>21</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your game with the Nike Air Max. Desig...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
    </tr>
    <tr>
      <th>22</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>23</th>
      <td>Organic Almond Butter</td>
      <td>Food</td>
      <td>Indulge in the creamy goodness of Organic Almo...</td>
      <td>[-0.014613640494644642, -0.002179765608161688,...</td>
    </tr>
    <tr>
      <th>24</th>
      <td>Yamaha YZF-R3</td>
      <td>Mobile Phones</td>
      <td>Introducing the Yamaha YZF-R3, the ultimate sp...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
    </tr>
    <tr>
      <th>25</th>
      <td>Adidas Ultraboost</td>
      <td>Shoes</td>
      <td>Discover the Adidas Ultraboost, a shoe that of...</td>
      <td>[0.03944042697548866, 0.022062409669160843, -0...</td>
    </tr>
    <tr>
      <th>26</th>
      <td>Philips Sonicare</td>
      <td>Electronic Toothbrush</td>
      <td>Experience the dental care revolution with Phi...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>27</th>
      <td>Organic Quinoa</td>
      <td>Food</td>
      <td>Nourish your body with Organic Quinoa, a nutri...</td>
      <td>[-0.014613640494644642, -0.002179765608161688,...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "42a1def6-820a-4294-ad5e-9b733a261a0f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "affff5c6-4a1f-49d4-953b-67dd05ec25e0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "aae4ded9088f0dfe449517050ed569c7ca743825f9a35924eed5080eb1a89db5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "768179ab-7d6e-445b-9e0e-03480b7ebde5", "node_type": "1", "metadata": {}, "hash": "519a5446b5d6e76792329059674cb699b4dcea207584a004c9e5bb695cb12a2f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33091, "end_char_idx": 35530, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now we perform the elbow method. 


```python
# Determine the optimal number of clusters using the elbow method
inertias = []
range_of_clusters = range(1, 13)  # Adjust the range as necessary

for n_clusters in range_of_clusters:
    kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42, n_init=10)
    kmeans.fit(matrix)
    inertias.append(kmeans.inertia_)

```

This will output a chart for us in which we have to visually tell where the optimal cluster point is. We can see below that we see a gradual decrease of inertia rather than a sharp elbow but the point of steepest decrease appears to occur around 3, 4 or 5 clusters which lines up with our expectations given our prompt. 


```python
# Plotting the elbow plot
plt.figure(figsize=(10, 6))
plt.plot(range_of_clusters, inertias, '-o')
plt.title('Elbow Method to Determine Optimal Number of Clusters')
plt.xlabel('Number of Clusters')
plt.ylabel('Inertia')
plt.xticks(range_of_clusters)
plt.show()
```


    
![png](output_32_0.png)
    


![elbow_chart](../images/elbow_chart.png)

For demonstration purposes we will pick 5 as the optimal cluster number to show it doesn't matter exactly where we pick it as long as we are approximately right. There are numerous correct ways to categorize data. We also store which cluster each data point belongs to.


```python
n_clusters = 5

kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42)
kmeans.fit(matrix)
labels = kmeans.labels_
df["Cluster"] = labels
```

We will analyze the cluster data now. There are two separate things we will look to address. 1. imbalanced data, 2. Expanding the data distribution.

First for imbalanced data we count the number of examples in each cluster. Then we select a few examples from each cluster at random and ask the LLM what topics these map to. 


```python
cluster_counts = df["Cluster"].value_counts().sort_index()
print(cluster_counts)
```

    Cluster
    0    5
    1    7
    2    8
    3    6
    4    2
    Name: count, dtype: int64


We can see the topics found here:
Eco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel
match well enough but not exactly to our initial prompt of:
vehicle, clothing, toiletries, food.

As we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.


```python
df
```


key: _node_content
string_value:
{"id_": "768179ab-7d6e-445b-9e0e-03480b7ebde5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "42a1def6-820a-4294-ad5e-9b733a261a0f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cc9044e103aed33854746200760770356f9de0a071074c663a7984ae56bac56a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "721c4fb9-f07a-4ee7-bf05-e79124ac51b5", "node_type": "1", "metadata": {}, "hash": "ab588b90d6dc24973fdb2a10671cbe85c01658c06925f07939fd38007b93dadf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34080, "end_char_idx": 36391, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![png](output_32_0.png)
    


![elbow_chart](../images/elbow_chart.png)

For demonstration purposes we will pick 5 as the optimal cluster number to show it doesn't matter exactly where we pick it as long as we are approximately right. There are numerous correct ways to categorize data. We also store which cluster each data point belongs to.


```python
n_clusters = 5

kmeans = KMeans(n_clusters=n_clusters, init="k-means++", random_state=42)
kmeans.fit(matrix)
labels = kmeans.labels_
df["Cluster"] = labels
```

We will analyze the cluster data now. There are two separate things we will look to address. 1. imbalanced data, 2. Expanding the data distribution.

First for imbalanced data we count the number of examples in each cluster. Then we select a few examples from each cluster at random and ask the LLM what topics these map to. 


```python
cluster_counts = df["Cluster"].value_counts().sort_index()
print(cluster_counts)
```

    Cluster
    0    5
    1    7
    2    8
    3    6
    4    2
    Name: count, dtype: int64


We can see the topics found here:
Eco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel
match well enough but not exactly to our initial prompt of:
vehicle, clothing, toiletries, food.

As we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
      <th>Cluster</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max.


key: _node_content
string_value:
{"id_": "721c4fb9-f07a-4ee7-bf05-e79124ac51b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "768179ab-7d6e-445b-9e0e-03480b7ebde5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2688c9dc0c2bc7561472fe3f06bf65a10502f11b56b9627b55efc4a3f4c84bdb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0", "node_type": "1", "metadata": {}, "hash": "f14180b8bdbd0b4c68e75ec73f6c4b0666f7dfe6bc3f34e8971b82290db22811", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35120, "end_char_idx": 37346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We can see the topics found here:
Eco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel
match well enough but not exactly to our initial prompt of:
vehicle, clothing, toiletries, food.

As we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
      <th>Cluster</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max. C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans.


key: _node_content
string_value:
{"id_": "ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "721c4fb9-f07a-4ee7-bf05-e79124ac51b5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f42935f28f352748287e1c7611591b620a43558e297a3eb3bbd3a2e92a9d1a42", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a9aecde-b131-49c8-bc9e-f8d9a642957b", "node_type": "1", "metadata": {}, "hash": "f1add170365f3379c540b1f73b5ada90f796f3a47fb5d8dd05e996f140f33097", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 36392, "end_char_idx": 37842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans. Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal.


key: _node_content
string_value:
{"id_": "8a9aecde-b131-49c8-bc9e-f8d9a642957b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e1d43104c8f95caa7d87bed0e13e52743e28644f7439a15bd84752cdc1f41985", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2dc16581-17fe-4db6-810a-d7a9f2f489e1", "node_type": "1", "metadata": {}, "hash": "99a44190d96225c11abbd319e7096947f4d857f95b2208bcb53cf232008e8975", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37347, "end_char_idx": 39286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal. This...</td>
      <td>[-0.006529285106807947, 0.007865572348237038, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Toyota Camry</td>
      <td>Sedan</td>
      <td>The Toyota Camry stands out in the sedan categ...</td>
      <td>[-0.02088991366326809, -0.006191295105963945, ...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Adidas Ultraboost</td>
      <td>Running Shoes</td>
      <td>Run like never before in the Adidas Ultraboost...</td>
      <td>[0.02679188922047615, 0.014639599248766899, 8....</td>
      <td>2</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Toyota Camry</td>
      <td>Car</td>
      <td>The Toyota Camry is a reliable midsize sedan k...</td>
      <td>[0.008056452497839928, -0.007912316359579563, ...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your sneaker game with the Nike Air Ma...</td>
      <td>[0.03943241760134697, 0.02208484522998333, -0....</td>
      <td>2</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Colgate Electric Toothbrush</td>
      <td>Electronic Toothbrush</td>
      <td>Transform your oral hygiene routine with the C...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>13</th>
      <td>Blue Diamond Almonds</td>
      <td>Nuts</td>
      <td>Snack healthy with Blue Diamond Almonds.


key: _node_content
string_value:
{"id_": "2dc16581-17fe-4db6-810a-d7a9f2f489e1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a9aecde-b131-49c8-bc9e-f8d9a642957b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1eeb7c90ed25fd846ba4991cd7e0fc05299c849fcf8279081a38dc5857645d43", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae4a36c3-8ff9-417e-8763-e7e8e140fe79", "node_type": "1", "metadata": {}, "hash": "124b692da8879be5ce1feb8eb7aba87ab0552b2618ee37d9342b69d7eed3982d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39287, "end_char_idx": 41195, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These...</td>
      <td>[-0.013289917260408401, 0.036334190517663956, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>14</th>
      <td>Harley Davidson Fat Boy</td>
      <td>Motorcycle</td>
      <td>Experience the thrill of the open road with th...</td>
      <td>[0.012365399859845638, 0.03552943095564842, -0...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>15</th>
      <td>Adidas UltraBoost</td>
      <td>Sneakers</td>
      <td>Enjoy a perfect blend of comfort and performan...</td>
      <td>[0.013107392005622387, 0.02963760495185852, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>16</th>
      <td>Dove Men's Body Wash</td>
      <td>Body Wash</td>
      <td>Refresh and hydrate your skin with Dove Men's ...</td>
      <td>[0.03760576993227005, -0.008475445210933685, -...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>17</th>
      <td>Quaker Oats</td>
      <td>Oats</td>
      <td>Start your day right with Quaker Oats. Packed ...</td>
      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>
      <td>3</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Ford F-150</td>
      <td>Truck</td>
      <td>The Ford F-150 is a durable and dependable tru...</td>
      <td>[0.023461222648620605, -0.026651185005903244, ...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>19</th>
      <td>Levi's 501 Jeans</td>
      <td>Jeans</td>
      <td>Discover the timeless style of Levi's 501 Jean...</td>
      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>20</th>
      <td>Tesla Model 3</td>
      <td>Mobile Phones</td>
      <td>Explore the future of driving with the Tesla M...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
      <td>4</td>
    </tr>
    <tr>
      <th>21</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your game with the Nike Air Max.


key: _node_content
string_value:
{"id_": "ae4a36c3-8ff9-417e-8763-e7e8e140fe79", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2dc16581-17fe-4db6-810a-d7a9f2f489e1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ac21526764b689f3d07f7616a439ea7dd8af619c38e86e2b566c45e8989c0ef8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68ea9042-1ab1-4165-ad2d-6ca66decbf5e", "node_type": "1", "metadata": {}, "hash": "368f9d8f3cc867d0c3d437f4817edfabfb6261b4c844562baa75cd77554cedc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40251, "end_char_idx": 42788, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Packed ...</td>
      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>
      <td>3</td>
    </tr>
    <tr>
      <th>18</th>
      <td>Ford F-150</td>
      <td>Truck</td>
      <td>The Ford F-150 is a durable and dependable tru...</td>
      <td>[0.023461222648620605, -0.026651185005903244, ...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>19</th>
      <td>Levi's 501 Jeans</td>
      <td>Jeans</td>
      <td>Discover the timeless style of Levi's 501 Jean...</td>
      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>20</th>
      <td>Tesla Model 3</td>
      <td>Mobile Phones</td>
      <td>Explore the future of driving with the Tesla M...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
      <td>4</td>
    </tr>
    <tr>
      <th>21</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your game with the Nike Air Max. Desig...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>22</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>23</th>
      <td>Organic Almond Butter</td>
      <td>Food</td>
      <td>Indulge in the creamy goodness of Organic Almo...</td>
      <td>[-0.014613640494644642, -0.002179765608161688,...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>24</th>
      <td>Yamaha YZF-R3</td>
      <td>Mobile Phones</td>
      <td>Introducing the Yamaha YZF-R3, the ultimate sp...</td>
      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>
      <td>4</td>
    </tr>
    <tr>
      <th>25</th>
      <td>Adidas Ultraboost</td>
      <td>Shoes</td>
      <td>Discover the Adidas Ultraboost, a shoe that of...</td>
      <td>[0.03944042697548866, 0.022062409669160843, -0...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>26</th>
      <td>Philips Sonicare</td>
      <td>Electronic Toothbrush</td>
      <td>Experience the dental care revolution with Phi...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>27</th>
      <td>Organic Quinoa</td>
      <td>Food</td>
      <td>Nourish your body with Organic Quinoa, a nutri...</td>
      <td>[-0.014613640494644642, -0.002179765608161688,...</td>
      <td>3</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "68ea9042-1ab1-4165-ad2d-6ca66decbf5e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae4a36c3-8ff9-417e-8763-e7e8e140fe79", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a99ff1f361285b3caa5cec8fb5657e1e5ae3059421b36e1cf1660747cab0d5a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ed3a3c0-7464-445f-82af-536e7712c0ad", "node_type": "1", "metadata": {}, "hash": "3900616795835eff1cc188894c127aee6d00eb44cf45ea69c8a07017b01bb124", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42793, "end_char_idx": 45309, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
selected_examples = df.groupby('Cluster').apply(lambda x: x.sample(3, replace=True)).reset_index(drop=True)

# Format the selected examples
formatted_examples = "\n".join(
    f'Input: "{row["Product"]}, {row["Category"]}"\nOutput: "{row["Description"]}"\nCluster: "{row["Cluster"]}"'
    for _, row in selected_examples.iterrows()
)

topic_prompt = f"""
    I previously generated some examples of input output trainings pairs and then I clustered them based on category. From each cluster I picked 3 example data point which you can find below.
    I want you identify the broad topic areas these clusters belong to.
    Previous examples:
    {formatted_examples}


    Your output should be strictly of the format:
    Cluster: number, topic: topic
    Cluster: number, topic: topic
    Cluster: number, topic: topic

    Do not add any extra characters around that formatting as it will make the output parsing break.
    """

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed analyze clustered data"},
    {"role": "user", "content": topic_prompt}
  ]
)
res = response.choices[0].message.content

pattern = r"Cluster: (\d+), topic: ([^\n]+)"
matches = re.findall(pattern, res)
clusters = [{"cluster": int(cluster), "topic": topic} for cluster, topic in matches]
json_output = json.dumps(clusters, indent=2)
print(json_output)
```

    [
      {
        "cluster": 0,
        "topic": "Automotive  "
      },
      {
        "cluster": 1,
        "topic": "Personal Care  "
      },
      {
        "cluster": 2,
        "topic": "Footwear  "
      },
      {
        "cluster": 3,
        "topic": "Food  "
      },
      {
        "cluster": 4,
        "topic": "Automotive  "
      }
    ]


We now have the clusters and their counts so we could prompt the LLM to generate more examples within the topics we want. However for this example we won't take that further as they are well-split and you would just follow the procedure above for prompting the model to generate data while passing in the underrepresented topics.

Next, we will try and deal with increasing the diversity of our data distribution. 

First we start in a similar way by finding a few examples from each cluster at random and ask the LLM what topics these map to. In addition to this in the same LLM call, we will ask it to generate more topics to increase the diversity of our data. We do this in one call to save time/cost.


key: _node_content
string_value:
{"id_": "3ed3a3c0-7464-445f-82af-536e7712c0ad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68ea9042-1ab1-4165-ad2d-6ca66decbf5e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e5e864e3ccc14741db51c7a3c776da99a72b550b164b1dcce72e3ab3fdc2243f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e83cfcac-b5fe-4174-9f11-100ca390c386", "node_type": "1", "metadata": {}, "hash": "e00b36a5841e2b99f4bcdb2851afcb5c7b919ef4667b4f1999dc60a825aafc48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 44604, "end_char_idx": 47554, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We now have the clusters and their counts so we could prompt the LLM to generate more examples within the topics we want. However for this example we won't take that further as they are well-split and you would just follow the procedure above for prompting the model to generate data while passing in the underrepresented topics.

Next, we will try and deal with increasing the diversity of our data distribution. 

First we start in a similar way by finding a few examples from each cluster at random and ask the LLM what topics these map to. In addition to this in the same LLM call, we will ask it to generate more topics to increase the diversity of our data. We do this in one call to save time/cost.


```python
selected_examples = df.groupby('Cluster').apply(lambda x: x.sample(3, replace=True)).reset_index(drop=True)

# Format the selected examples
formatted_examples = "\n".join(
    f'Input: "{row["Product"]}, {row["Category"]}"\nOutput: "{row["Description"]}"\nCluster: "{row["Cluster"]}"'
    for _, row in selected_examples.iterrows()
)

topic_prompt = f"""
    I previously generated some examples of input output trainings pairs and then I clustered them based on category. From each cluster I picked 3 example data point which you can find below.
    I want to promote diversity in my examples across categories so follow the procedure below:
    1. You must identify the broad topic areas these clusters belong to.
    2. You should generate further topic areas which don't exist so I can generate data within these topics to improve diversity.


    Previous examples:
    {formatted_examples}


    Your output should be strictly of the format:

    1. Cluster topic mapping
    Cluster: number, topic: topic
    Cluster: number, topic: topic
    Cluster: number, topic: topic

    2. New topics
    1. topic
    2. topic
    3. topic
    4. topic

    Do not add any extra characters around that formatting as it will make the output parsing break. It is very important you stick to that output format
    """

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to analyze clustered data"},
    {"role": "user", "content": topic_prompt}
  ]
)
res = response.choices[0].message.content
print(res)

```

    1. Cluster topic mapping
    Cluster: 0, topic: Automotive
    Cluster: 1, topic: Personal Care
    Cluster: 2, topic: Footwear
    Cluster: 3, topic: Food
    Cluster: 4, topic: Electric Vehicles
    
    2. New topics
    1. topic: Home Appliances
    2. topic: Outdoor Equipment
    3. topic: Smart Home Technology
    4. topic: Fitness Equipment


We can see here again that we explicitly prompt the output structure it should follow. I also tell it the purpose of generating topics (to promote diversity) so the model has full context.

We then parse the data into a list of cluster-mapping jsons and a list of topics


key: _node_content
string_value:
{"id_": "e83cfcac-b5fe-4174-9f11-100ca390c386", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ed3a3c0-7464-445f-82af-536e7712c0ad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8f416ceaed809b0c9ad4049b870cf96f42b47c0091d1d97dc89b974915bc1cb8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29badccb-8e7d-4004-9b0a-127e0d5df127", "node_type": "1", "metadata": {}, "hash": "5e12a56e068e52536d2aca75818cf3d00e2f40a8d99d185b6582d72931d9feb9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 46174, "end_char_idx": 49208, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Previous examples:
    {formatted_examples}


    Your output should be strictly of the format:

    1. Cluster topic mapping
    Cluster: number, topic: topic
    Cluster: number, topic: topic
    Cluster: number, topic: topic

    2. New topics
    1. topic
    2. topic
    3. topic
    4. topic

    Do not add any extra characters around that formatting as it will make the output parsing break. It is very important you stick to that output format
    """

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to analyze clustered data"},
    {"role": "user", "content": topic_prompt}
  ]
)
res = response.choices[0].message.content
print(res)

```

    1. Cluster topic mapping
    Cluster: 0, topic: Automotive
    Cluster: 1, topic: Personal Care
    Cluster: 2, topic: Footwear
    Cluster: 3, topic: Food
    Cluster: 4, topic: Electric Vehicles
    
    2. New topics
    1. topic: Home Appliances
    2. topic: Outdoor Equipment
    3. topic: Smart Home Technology
    4. topic: Fitness Equipment


We can see here again that we explicitly prompt the output structure it should follow. I also tell it the purpose of generating topics (to promote diversity) so the model has full context.

We then parse the data into a list of cluster-mapping jsons and a list of topics


```python
parts = res.split("\n\n")
cluster_mapping_part = parts[0]
new_topics_part = parts[1]

# Parse cluster topic mapping
cluster_topic_mapping_lines = cluster_mapping_part.split("\n")[1:]  # Skip the first two lines
cluster_topic_mapping = [{"cluster": int(line.split(",")[0].split(":")[1].strip()), "topic": line.split(":")[2].strip()} for line in cluster_topic_mapping_lines]

# Parse new topics
new_topics_lines = new_topics_part.split("\n")[1:]  # Skip the first line
new_topics = [line.split(". ")[1] for line in new_topics_lines]

cluster_topic_mapping, new_topics
```




    ([{'cluster': 0, 'topic': 'Automotive'},
      {'cluster': 1, 'topic': 'Personal Care'},
      {'cluster': 2, 'topic': 'Footwear'},
      {'cluster': 3, 'topic': 'Food'},
      {'cluster': 4, 'topic': 'Electric Vehicles'}],
     ['topic: Home Appliances',
      'topic: Outdoor Equipment',
      'topic: Smart Home Technology',
      'topic: Fitness Equipment'])



And finally we can use this information to further prompt a model to keep generating synthetic data. We do this by passing all the topics in the list of jsons to the prompt below.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. I want the input to be product name and category and output to be description. the category should be things like: mobile phones, shoes, headphones, laptop, electronic toothbrush, etc. and also more importantly the categories should come under some main topics: {[entry['topic'] for entry in cluster_topic_mapping]})
  After the number of each example also state the topic area.


key: _node_content
string_value:
{"id_": "29badccb-8e7d-4004-9b0a-127e0d5df127", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e83cfcac-b5fe-4174-9f11-100ca390c386", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7de7a98c3a4df3f9bf2b4cf28aa970e2b26e3ba59d3d16a708769d4eb900da59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44017f70-3e39-4bf9-a2a2-9cdfebc244b6", "node_type": "1", "metadata": {}, "hash": "6e164adb9c4a0cce48857b0f5e7e7f1e6b4954fe9563a32abdb5aa07bfa0bc60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 48145, "end_char_idx": 51288, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
([{'cluster': 0, 'topic': 'Automotive'},
      {'cluster': 1, 'topic': 'Personal Care'},
      {'cluster': 2, 'topic': 'Footwear'},
      {'cluster': 3, 'topic': 'Food'},
      {'cluster': 4, 'topic': 'Electric Vehicles'}],
     ['topic: Home Appliances',
      'topic: Outdoor Equipment',
      'topic: Smart Home Technology',
      'topic: Fitness Equipment'])



And finally we can use this information to further prompt a model to keep generating synthetic data. We do this by passing all the topics in the list of jsons to the prompt below.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. I want the input to be product name and category and output to be description. the category should be things like: mobile phones, shoes, headphones, laptop, electronic toothbrush, etc. and also more importantly the categories should come under some main topics: {[entry['topic'] for entry in cluster_topic_mapping]})
  After the number of each example also state the topic area. The format should be of the form:
  1. topic_area
  Input: product_name, category
  Output: description

  Do not add any extra characters around that formatting as it will make the output parsing break.

  Here are some helpful examples so you get the style of output correct.

  1) clothing
  Input: "Shoe Name, Shoes"
  Output: "Experience unparalleled comfort. These shoes feature a blend of modern style and the traditional superior cushioning, perfect for those always on the move."
  """

  response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string)
```

    1. Automotive  
    Input: "Tesla Model S, Electric Vehicles"  
    Output: "The Tesla Model S delivers exhilarating performance with advanced electric technology, offering a sleek design, impressive range, and an industry-leading infotainment system."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "The Oral-B Pro 1000 features a 3D cleaning action that oscillates, rotates, and pulsates to remove plaque, ensuring a deeper clean for healthier gums."
    
    3. Footwear  
    Input: "Nike Air Max 270, Shoes"  
    Output: "Step into comfort and style with Nike Air Max 270, designed with a large Max Air unit for superior cushioning and a breathable upper for a snug fit."
    
    4. Electronics  
    Input: "Apple iPhone 12, Mobile Phones"  
    Output: "The Apple iPhone 12 combines powerful performance with stunning design, equipped with A14 Bionic chip and advanced camera systems for capturing every moment in stunning detail."
    
    5. Food  
    Input: "Nature Valley Granola Bars, Snacks"  
    Output: "Nature Valley Granola Bars offer a wholesome crunch made from simple, delicious ingredients, providing a perfect snack that fuels your adventure."
    
    6.


key: _node_content
string_value:
{"id_": "44017f70-3e39-4bf9-a2a2-9cdfebc244b6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29badccb-8e7d-4004-9b0a-127e0d5df127", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "372b86e7634604eadf3a97c5926214ba8e56cd96d9a7b8554613bbe2df1ae313", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b715bbb3-9dd8-4d28-8724-1830110527a3", "node_type": "1", "metadata": {}, "hash": "35b949946161073f3b5dc4e7645c86feaae92413cf584bdd5fecb01a5a632fc8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49700, "end_char_idx": 53046, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"""

  response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string)
```

    1. Automotive  
    Input: "Tesla Model S, Electric Vehicles"  
    Output: "The Tesla Model S delivers exhilarating performance with advanced electric technology, offering a sleek design, impressive range, and an industry-leading infotainment system."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "The Oral-B Pro 1000 features a 3D cleaning action that oscillates, rotates, and pulsates to remove plaque, ensuring a deeper clean for healthier gums."
    
    3. Footwear  
    Input: "Nike Air Max 270, Shoes"  
    Output: "Step into comfort and style with Nike Air Max 270, designed with a large Max Air unit for superior cushioning and a breathable upper for a snug fit."
    
    4. Electronics  
    Input: "Apple iPhone 12, Mobile Phones"  
    Output: "The Apple iPhone 12 combines powerful performance with stunning design, equipped with A14 Bionic chip and advanced camera systems for capturing every moment in stunning detail."
    
    5. Food  
    Input: "Nature Valley Granola Bars, Snacks"  
    Output: "Nature Valley Granola Bars offer a wholesome crunch made from simple, delicious ingredients, providing a perfect snack that fuels your adventure."
    
    6. Automotive  
    Input: "Ford F-150, Electric Vehicles"  
    Output: "The Ford F-150 stands at the forefront of durability and innovation, with its powerful electric version setting new standards for strength and sustainability in the truck category." 
    
    7. Personal Care  
    Input: "Philips Sonicare, Electronic Toothbrush"  
    Output: "Philips Sonicare delivers superior cleaning with dynamic technology that provides up to 31,000 strokes per minute for a healthier mouth and brighter smile."
    
    8. Footwear  
    Input: "Adidas Ultraboost, Shoes"  
    Output: "The Adidas Ultraboost is a game-changer in running footwear, featuring responsive cushioning and a knit upper for a snug, supportive fit that adapts to any run."
    
    9. Electronics  
    Input: "Dell XPS 13, Laptop"  
    Output: "The Dell XPS 13 is a remarkable laptop with an ultra-thin design, featuring a stunning InfinityEdge display and powerful performance to accommodate your multitasking needs."
    
    10. Food  
    Input: "Kraft Macaroni & Cheese, Instant Food"  
    Output: "Kraft Macaroni & Cheese offers quick and convenient comfort food, combining creamy cheese sauce with perfectly cooked pasta for a simple meal that satisfies."
    
    1. Automotive  
    Input: "Toyota Camry, Mobile Phones"  
    Output: "The Toyota Camry is a midsize sedan that combines efficiency with modern technology. It offers a spacious interior and the latest features for an enjoyable driving experience."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "The Oral-B Pro 1000 not only provides powerful cleaning action but also enhances your oral hygiene routine with its smart pressure sensor and various cleaning modes."


key: _node_content
string_value:
{"id_": "b715bbb3-9dd8-4d28-8724-1830110527a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44017f70-3e39-4bf9-a2a2-9cdfebc244b6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2b45a9241aa99d664bfec87ddb60fffeffc4026bbeaa7879a8d1059a1956a2bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "478a64ba-97d4-4f5e-b39b-e6a02e193db9", "node_type": "1", "metadata": {}, "hash": "06d20558ec197ece5dfe94b4df1f7d7bbc2c9a78349e6db380efaaec7480cf36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51289, "end_char_idx": 54701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Automotive  
    Input: "Ford F-150, Electric Vehicles"  
    Output: "The Ford F-150 stands at the forefront of durability and innovation, with its powerful electric version setting new standards for strength and sustainability in the truck category." 
    
    7. Personal Care  
    Input: "Philips Sonicare, Electronic Toothbrush"  
    Output: "Philips Sonicare delivers superior cleaning with dynamic technology that provides up to 31,000 strokes per minute for a healthier mouth and brighter smile."
    
    8. Footwear  
    Input: "Adidas Ultraboost, Shoes"  
    Output: "The Adidas Ultraboost is a game-changer in running footwear, featuring responsive cushioning and a knit upper for a snug, supportive fit that adapts to any run."
    
    9. Electronics  
    Input: "Dell XPS 13, Laptop"  
    Output: "The Dell XPS 13 is a remarkable laptop with an ultra-thin design, featuring a stunning InfinityEdge display and powerful performance to accommodate your multitasking needs."
    
    10. Food  
    Input: "Kraft Macaroni & Cheese, Instant Food"  
    Output: "Kraft Macaroni & Cheese offers quick and convenient comfort food, combining creamy cheese sauce with perfectly cooked pasta for a simple meal that satisfies."
    
    1. Automotive  
    Input: "Toyota Camry, Mobile Phones"  
    Output: "The Toyota Camry is a midsize sedan that combines efficiency with modern technology. It offers a spacious interior and the latest features for an enjoyable driving experience."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "The Oral-B Pro 1000 not only provides powerful cleaning action but also enhances your oral hygiene routine with its smart pressure sensor and various cleaning modes."
    
    3. Footwear  
    Input: "Nike Air Max, Shoes"  
    Output: "Step into comfort with the Nike Air Max. With cutting-edge technology and a sleek design, these shoes are perfect for athletes and casual wearers alike."
    
    4. Food  
    Input: "Nature's Valley Granola Bar, Food"  
    Output: "Savor the wholesome goodness of Nature's Valley Granola Bar, crafted with real ingredients to fuel your day with delicious flavor and crunchy satisfaction."
    
    5. Electric Vehicles  
    Input: "Tesla Model 3, Mobile Phones"  
    Output: "The Tesla Model 3 is a revolutionary electric vehicle that combines performance with sustainability, featuring an intuitive interface and cutting-edge technology for an exceptional driving experience."
    
    1. Automotive  
    Input: "Tesla Model 3, Electric Vehicles"  
    Output: "The Tesla Model 3 combines cutting-edge technology with eco-friendly driving. Enjoy a sleek design, impressive range, and top-notch safety features, making it the perfect electric car for the modern driver."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "Achieve a superior clean with the Oral-B Pro 1000. Featuring advanced 3D cleaning action, this electronic toothbrush ensures effective plaque removal while being gentle on gums, allowing you to maintain optimum oral health."
    
    3. Footwear  
    Input: "Nike Air Max, Shoes"  
    Output: "Step up your game with Nike Air Max shoes. Combining iconic cushioning technology and bold style, these shoes provide ultimate comfort and support, perfect for both casual wear and athletic performance."
    
    4.


key: _node_content
string_value:
{"id_": "478a64ba-97d4-4f5e-b39b-e6a02e193db9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b715bbb3-9dd8-4d28-8724-1830110527a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "22f79412780ca41d5afcb4fad61c91b72657cb9b5b954ae0cc453d69225ae8ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77166dbd-4193-40c8-95bb-698620c34315", "node_type": "1", "metadata": {}, "hash": "ba6c7a43bbb429498869a4127b92aaef77686426698e0e8d047d28bb3c1f6c70", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53056, "end_char_idx": 56419, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3. Footwear  
    Input: "Nike Air Max, Shoes"  
    Output: "Step into comfort with the Nike Air Max. With cutting-edge technology and a sleek design, these shoes are perfect for athletes and casual wearers alike."
    
    4. Food  
    Input: "Nature's Valley Granola Bar, Food"  
    Output: "Savor the wholesome goodness of Nature's Valley Granola Bar, crafted with real ingredients to fuel your day with delicious flavor and crunchy satisfaction."
    
    5. Electric Vehicles  
    Input: "Tesla Model 3, Mobile Phones"  
    Output: "The Tesla Model 3 is a revolutionary electric vehicle that combines performance with sustainability, featuring an intuitive interface and cutting-edge technology for an exceptional driving experience."
    
    1. Automotive  
    Input: "Tesla Model 3, Electric Vehicles"  
    Output: "The Tesla Model 3 combines cutting-edge technology with eco-friendly driving. Enjoy a sleek design, impressive range, and top-notch safety features, making it the perfect electric car for the modern driver."
    
    2. Personal Care  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "Achieve a superior clean with the Oral-B Pro 1000. Featuring advanced 3D cleaning action, this electronic toothbrush ensures effective plaque removal while being gentle on gums, allowing you to maintain optimum oral health."
    
    3. Footwear  
    Input: "Nike Air Max, Shoes"  
    Output: "Step up your game with Nike Air Max shoes. Combining iconic cushioning technology and bold style, these shoes provide ultimate comfort and support, perfect for both casual wear and athletic performance."
    
    4. Food  
    Input: "Oreo Cookies, Snacks"  
    Output: "Indulge in the classic taste of Oreo Cookies. With their irresistible cream filling sandwiched between two crunchy chocolate wafers, these treats are perfect for satisfying your sweet tooth any time of the day."
    
    5. Personal Care  
    Input: "Garnier Micellar Water, Skincare"  
    Output: "Garnier Micellar Water gently removes makeup and impurities while hydrating the skin. This soothing formula is suitable for all skin types, making it a must-have in your daily skincare routine."
    
    6. Automotive  
    Input: "Ford F-150, Trucks"  
    Output: "The Ford F-150 is the quintessential pickup truck, combining power, reliability, and innovative technology. Equipped with advanced towing capabilities and a spacious interior, it's designed for both work and play."
    
    7. Electronics  
    Input: "Samsung Galaxy S21, Mobile Phones"  
    Output: "Experience the future of mobile technology with the Samsung Galaxy S21. This smartphone features a stunning display, powerful processor, and multiple camera options, perfect for capturing life's moments in high definition."
    
    8. Footwear  
    Input: "Adidas Ultraboost, Shoes"  
    Output: "Run in style with Adidas Ultraboost shoes. Known for their comfort and performance, these shoes utilize responsive cushioning to provide unmatched energy return with every step you take." 
    
    9. Electronics  
    Input: "Dell XPS 13, Laptops"  
    Output: "The Dell XPS 13 redefines the laptop experience with its stunning InfinityEdge display, powerful performance, and sleek design. Ideal for both professionals and students looking for portability and functionality."
    
    10.


key: _node_content
string_value:
{"id_": "77166dbd-4193-40c8-95bb-698620c34315", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "478a64ba-97d4-4f5e-b39b-e6a02e193db9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "94c552fbaaaa3ea15979341f0620730679e48cc1937546eb7104bffd2183b90d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 54699, "end_char_idx": 57393, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4. Food  
    Input: "Oreo Cookies, Snacks"  
    Output: "Indulge in the classic taste of Oreo Cookies. With their irresistible cream filling sandwiched between two crunchy chocolate wafers, these treats are perfect for satisfying your sweet tooth any time of the day."
    
    5. Personal Care  
    Input: "Garnier Micellar Water, Skincare"  
    Output: "Garnier Micellar Water gently removes makeup and impurities while hydrating the skin. This soothing formula is suitable for all skin types, making it a must-have in your daily skincare routine."
    
    6. Automotive  
    Input: "Ford F-150, Trucks"  
    Output: "The Ford F-150 is the quintessential pickup truck, combining power, reliability, and innovative technology. Equipped with advanced towing capabilities and a spacious interior, it's designed for both work and play."
    
    7. Electronics  
    Input: "Samsung Galaxy S21, Mobile Phones"  
    Output: "Experience the future of mobile technology with the Samsung Galaxy S21. This smartphone features a stunning display, powerful processor, and multiple camera options, perfect for capturing life's moments in high definition."
    
    8. Footwear  
    Input: "Adidas Ultraboost, Shoes"  
    Output: "Run in style with Adidas Ultraboost shoes. Known for their comfort and performance, these shoes utilize responsive cushioning to provide unmatched energy return with every step you take." 
    
    9. Electronics  
    Input: "Dell XPS 13, Laptops"  
    Output: "The Dell XPS 13 redefines the laptop experience with its stunning InfinityEdge display, powerful performance, and sleek design. Ideal for both professionals and students looking for portability and functionality."
    
    10. Personal Care  
    Input: "Philips Sonicare, Electronic Toothbrush"  
    Output: "Philips Sonicare's electronic toothbrush guarantees a superior cleaning experience with its advanced sonic technology. This toothbrush not only helps remove plaque but also promotes healthier gums for a brighter smile."
    
    


You can run this in a loop to append to your previous data and in this way you can keep generating more textual synthetic data to train another GPT model while making sure that we cater to imbalanced datasets and generating a diversity of data.

You have now completed part 1 of the synthetic data generation tutorial where we have gone through:
*   CSV with a structured prompt
*   CSV with a Python program
*   Multitable CSV with a python program
*   Simply creating textual data
*   Dealing with imbalanced or non-diverse textual data

In part 2 you will find find out techniques for better prompting an LLM to enhance textual synthetic data generation.


key: _node_content
string_value:
{"id_": "f7e81122-0e8b-4ae1-bf54-e7ccbe7d0d25", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fabcf30d-150d-4fb9-aa7d-808cfc9f3fae", "node_type": "1", "metadata": {}, "hash": "dfa65dae1b5e3305b0aa3cbd7ee1abad2219399c26154c7797fc2df3cf616163", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3512, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use guardrails

In this notebook we share examples of how to implement guardrails for your LLM applications. A guardrail is a generic term for **detective controls** that aim to steer your application. Greater steerability is a common requirement given the inherent randomness of LLMs, and so creating effective guardrails has become one of the most common areas of performance optimization when pushing an LLM from prototype to production. 

Guardrails are incredibly [diverse](https://github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/README.md) and can be deployed to virtually any context you can imagine something going wrong with LLMs. This notebook aims to give simple examples that can be extended to meet your unique use case, as well as outlining the trade-offs to consider when deciding whether to implement a guardrail, and how to do it.

This notebook will focus on:
1. **Input guardrails** that flag inappropriate content before it gets to your LLM
2. **Output guardrails** that validate what your LLM has produced before it gets to the customer

**Note:** This notebook tackles guardrails as a generic term for detective controls around an LLM - for the official libraries that provide distributions of pre-built guardrails frameworks, please check out the following:
- [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails/tree/main)
- [Guardrails AI](https://github.com/ShreyaR/guardrails)



```python
import openai

GPT_MODEL = 'gpt-4o-mini'
```

## 1. Input guardrails

Input guardrails aim to prevent inappropriate content getting to the LLM in the first place - some common use cases are:
- **Topical guardrails:** Identify when a user asks an off-topic question and give them advice on what topics the LLM can help them with.
- **Jailbreaking:** Detect when a user is trying to hijack the LLM and override its prompting.
- **Prompt injection:** Pick up instances of prompt injection where users try to hide malicious code that will be executed in any downstream functions the LLM executes. 

In all of these they act as a preventative control, running either before or in parallel with the LLM, and triggering your application to behave differently if one of these criteria are met.

### Designing a guardrail

When designing guardrails it is important to consider the trade-off between **accuracy**, **latency** and **cost**, where you try to achieve maximum accuracy for the least impact to your bottom line and the user's experience. 

We'll begin with a simple **topical guardrail** which aims to detect off-topic questions and prevent the LLM from answering if triggered. This guardrail consists of a simple prompt and uses `gpt-4o-mini`, maximising latency/cost holding a good enough accuracy, but if we wanted to optimize further we could consider:
- **Accuracy:** You could consider fine-tuning `gpt-4o-mini` or few-shot examples to increase the accuracy. RAG can also be effective if you have a corpus of information that can help determine whether a piece of content is allowed or not.
- **Latency/Cost:** You could try fine-tuning smaller models, such as `babbage-002` or open-source offerings like Llama, which can perform quite well when given enough training examples. When using open-source offerings you can also tune the machines you are using for inference to maximize either cost or latency reduction.

This simple guardrail aims to ensure the LLM only answers to a predefined set of topics, and responds to out-of-bounds queries with a canned message.


key: _node_content
string_value:
{"id_": "fabcf30d-150d-4fb9-aa7d-808cfc9f3fae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7e81122-0e8b-4ae1-bf54-e7ccbe7d0d25", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36ee9026eab3e5564534cebb660e84bccd878465ee688370d4ae212501105f70", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72221325-e558-4d1a-a47d-817b7a316071", "node_type": "1", "metadata": {}, "hash": "01d5271694fc3c539a2161ccddec0c0bfd6a285d726e60280e6bb0578aa7bac3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1767, "end_char_idx": 5400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- **Jailbreaking:** Detect when a user is trying to hijack the LLM and override its prompting.
- **Prompt injection:** Pick up instances of prompt injection where users try to hide malicious code that will be executed in any downstream functions the LLM executes. 

In all of these they act as a preventative control, running either before or in parallel with the LLM, and triggering your application to behave differently if one of these criteria are met.

### Designing a guardrail

When designing guardrails it is important to consider the trade-off between **accuracy**, **latency** and **cost**, where you try to achieve maximum accuracy for the least impact to your bottom line and the user's experience. 

We'll begin with a simple **topical guardrail** which aims to detect off-topic questions and prevent the LLM from answering if triggered. This guardrail consists of a simple prompt and uses `gpt-4o-mini`, maximising latency/cost holding a good enough accuracy, but if we wanted to optimize further we could consider:
- **Accuracy:** You could consider fine-tuning `gpt-4o-mini` or few-shot examples to increase the accuracy. RAG can also be effective if you have a corpus of information that can help determine whether a piece of content is allowed or not.
- **Latency/Cost:** You could try fine-tuning smaller models, such as `babbage-002` or open-source offerings like Llama, which can perform quite well when given enough training examples. When using open-source offerings you can also tune the machines you are using for inference to maximize either cost or latency reduction.

This simple guardrail aims to ensure the LLM only answers to a predefined set of topics, and responds to out-of-bounds queries with a canned message.

### Embrace async

A common design to minimize latency is to send your guardrails asynchronously along with your main LLM call. If your guardrails get triggered you send back their response, otherwise send back the LLM response.

We'll use this approach, creating an `execute_chat_with_guardrails` function that will run our LLM's `get_chat_response` and the `topical_guardrail` guardrail in parallel, and return the LLM response only if the guardrail returns `allowed`.

### Limitations

You should always consider the limitations of guardrails when developing your design. A few of the key ones to be aware of are:
- When using LLMs as a guardrail, be aware that they have the same vulnerabilities as your base LLM call itself. For example, a **prompt injection** attempt could be successful in evading both your guardrail and your actual LLM call.
- As conversations get longer, LLMs are more susceptible to **jailbreaking** as your instructions become diluted by the extra text.
- Guardrails can harm the user experience if you make them overly restrictive to compensate for the issues noted above. This manifests as **over-refusals**, where your guardrails reject innocuous user requests because there are similarities with prompt injection or jailbreaking attempts.

### Mitigations

If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.

We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails.


key: _node_content
string_value:
{"id_": "72221325-e558-4d1a-a47d-817b7a316071", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fabcf30d-150d-4fb9-aa7d-808cfc9f3fae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "be21df79ed339a4629b380a28f4599945e2c06044d39b9591439664a0ca44d0c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b787533-0ca9-4ad6-b3de-4acaa325880d", "node_type": "1", "metadata": {}, "hash": "c137f74e2e1f1c0e6efe9c3dcddcf768eacee5a4b4a986a276e7d955329100e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3514, "end_char_idx": 6613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Embrace async

A common design to minimize latency is to send your guardrails asynchronously along with your main LLM call. If your guardrails get triggered you send back their response, otherwise send back the LLM response.

We'll use this approach, creating an `execute_chat_with_guardrails` function that will run our LLM's `get_chat_response` and the `topical_guardrail` guardrail in parallel, and return the LLM response only if the guardrail returns `allowed`.

### Limitations

You should always consider the limitations of guardrails when developing your design. A few of the key ones to be aware of are:
- When using LLMs as a guardrail, be aware that they have the same vulnerabilities as your base LLM call itself. For example, a **prompt injection** attempt could be successful in evading both your guardrail and your actual LLM call.
- As conversations get longer, LLMs are more susceptible to **jailbreaking** as your instructions become diluted by the extra text.
- Guardrails can harm the user experience if you make them overly restrictive to compensate for the issues noted above. This manifests as **over-refusals**, where your guardrails reject innocuous user requests because there are similarities with prompt injection or jailbreaking attempts.

### Mitigations

If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.

We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails.


```python
system_prompt = "You are a helpful assistant."

bad_request = "I want to talk about horses"
good_request = "What are the best breeds of dog for people that like cats?"
```


```python
import asyncio


async def get_chat_response(user_request):
    print("Getting LLM response")
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_request},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=messages, temperature=0.5
    )
    print("Got LLM response")

    return response.choices[0].message.content


async def topical_guardrail(user_request):
    print("Checking topical guardrail")
    messages = [
        {
            "role": "system",
            "content": "Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'",
        },
        {"role": "user", "content": user_request},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=messages, temperature=0
    )

    print("Got guardrail response")
    return response.choices[0].message.content


key: _node_content
string_value:
{"id_": "0b787533-0ca9-4ad6-b3de-4acaa325880d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72221325-e558-4d1a-a47d-817b7a316071", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a1dcc370bc826a97359413dccd43974ea9d58fb48bf9cc3d3dd1acf99cedfe6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dc05992-3eb2-41f3-b337-fb3977b00730", "node_type": "1", "metadata": {}, "hash": "a992f8e782c8497123bc919d39f4f2136c90c581ce80031cad2dd46a8e96fda6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4787, "end_char_idx": 7543, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Mitigations

If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.

We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails.


```python
system_prompt = "You are a helpful assistant."

bad_request = "I want to talk about horses"
good_request = "What are the best breeds of dog for people that like cats?"
```


```python
import asyncio


async def get_chat_response(user_request):
    print("Getting LLM response")
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_request},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=messages, temperature=0.5
    )
    print("Got LLM response")

    return response.choices[0].message.content


async def topical_guardrail(user_request):
    print("Checking topical guardrail")
    messages = [
        {
            "role": "system",
            "content": "Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'",
        },
        {"role": "user", "content": user_request},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=messages, temperature=0
    )

    print("Got guardrail response")
    return response.choices[0].message.content


async def execute_chat_with_guardrail(user_request):
    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))
    chat_task = asyncio.create_task(get_chat_response(user_request))

    while True:
        done, _ = await asyncio.wait(
            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED
        )
        if topical_guardrail_task in done:
            guardrail_response = topical_guardrail_task.result()
            if guardrail_response == "not_allowed":
                chat_task.cancel()
                print("Topical guardrail triggered")
                return "I can only talk about cats and dogs, the best animals that ever lived."
            elif chat_task in done:
                chat_response = chat_task.result()
                return chat_response
        else:
            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again
```


key: _node_content
string_value:
{"id_": "8dc05992-3eb2-41f3-b337-fb3977b00730", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b787533-0ca9-4ad6-b3de-4acaa325880d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "29b0d615f6c5c8da04a7c03058e855688d9fd0553e2043de08779222a7c1d7c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1f04ad9-37a2-453e-bfa6-be4fca0c8808", "node_type": "1", "metadata": {}, "hash": "9e19fcc41be81fb1393d3127fe432725d65d4c932c0d8ab4711c3128c69285f2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6012, "end_char_idx": 9566, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
async def topical_guardrail(user_request):
    print("Checking topical guardrail")
    messages = [
        {
            "role": "system",
            "content": "Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'",
        },
        {"role": "user", "content": user_request},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=messages, temperature=0
    )

    print("Got guardrail response")
    return response.choices[0].message.content


async def execute_chat_with_guardrail(user_request):
    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))
    chat_task = asyncio.create_task(get_chat_response(user_request))

    while True:
        done, _ = await asyncio.wait(
            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED
        )
        if topical_guardrail_task in done:
            guardrail_response = topical_guardrail_task.result()
            if guardrail_response == "not_allowed":
                chat_task.cancel()
                print("Topical guardrail triggered")
                return "I can only talk about cats and dogs, the best animals that ever lived."
            elif chat_task in done:
                chat_response = chat_task.result()
                return chat_response
        else:
            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again
```


```python
# Call the main function with the good request - this should go through
response = await execute_chat_with_guardrail(good_request)
print(response)
```

    Checking topical guardrail
    Got guardrail response
    Getting LLM response
    Got LLM response
    If you like cats and are considering getting a dog, there are several breeds known for their compatibility with feline friends. Here are some of the best dog breeds that tend to get along well with cats:
    
    1. **Golden Retriever**: Friendly and tolerant, Golden Retrievers often get along well with other animals, including cats.
    
    2. **Labrador Retriever**: Similar to Golden Retrievers, Labs are social and friendly, making them good companions for cats.
    
    3. **Cavalier King Charles Spaniel**: This breed is gentle and affectionate, often forming strong bonds with other pets.
    
    4. **Basset Hound**: Basset Hounds are laid-back and generally have a calm demeanor, which can help them coexist peacefully with cats.
    
    5. **Beagle**: Beagles are friendly and sociable, and they often enjoy the company of other animals, including cats.
    
    6. **Pug**: Pugs are known for their playful and friendly nature, which can make them good companions for cats.
    
    7. **Shih Tzu**: Shih Tzus are typically friendly and adaptable, often getting along well with other pets.
    
    8. **Collie**: Collies are known for their gentle and protective nature, which can extend to their relationships with cats.
    
    9. **Newfoundland**: These gentle giants are known for their calm demeanor and often get along well with other animals.
    
    10. **Cocker Spaniel**: Cocker Spaniels are friendly and affectionate dogs that can get along well with cats if introduced properly.
    
    When introducing a dog to a cat, it's important to do so gradually and supervise their interactions to ensure a positive relationship. Each dog's personality can vary, so individual temperament is key in determining compatibility.


key: _node_content
string_value:
{"id_": "b1f04ad9-37a2-453e-bfa6-be4fca0c8808", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dc05992-3eb2-41f3-b337-fb3977b00730", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd688a516cb04677cd38df50f0f83036855992c2d30c9ffbf90d44683186415f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5f166be-ef95-489d-8fe2-eea89578b8d0", "node_type": "1", "metadata": {}, "hash": "3bccc3721b0040e85aa8c9c7dd1bc3611d817a2b5bc9067077ca12d3934b65dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9570, "end_char_idx": 13031, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Call the main function with the good request - this should get blocked
response = await execute_chat_with_guardrail(bad_request)
print(response)
```

    Checking topical guardrail
    Got guardrail response
    Getting LLM response
    Got LLM response
    Topical guardrail triggered
    I can only talk about cats and dogs, the best animals that ever lived.


Looks like our guardrail worked - the first question was allowed through, but the second was blocked for being off-topic. Now we'll extend this concept to moderate the response we get from the LLM as well.

## 2. Output guardrails

Output guardrails govern what the LLM comes back with. These can take many forms, with some of the most common being:
- **Hallucination/fact-checking guardrails:** Using a corpus of ground truth information or a training set of hallucinated responses to  block hallucinated responses.
- **Moderation guardrails:** Applying brand and corporate guidelines to moderate the LLM's results, and either blocking or rewriting its response if it breaches them.
- **Syntax checks:** Structured outputs from LLMs can be returned corrupt or unable to be parsed - these guardrails detect those and either retry or fail gracefully, preventing failures in downstream applications.
    - This is a common control to apply with function calling, ensuring that the expected schema is returned in the `arguments` when the LLM returns a `function_call`.
    
### Moderation guardrail

Here we implement a **moderation guardrail** that uses a version of the [G-Eval](https://arxiv.org/abs/2303.16634) evaluation method to score the presence of unwanted content in the LLM's response. This method is demonstrated in more detail in of our other [notebooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb).

To accomplish this we will make an extensible framework for moderating content that takes in a `domain` and applies `criteria` to a piece of `content` using a set of `steps`:
1. We set a domain name, which describes the type of content we're going to moderate.
2. We provide criteria, which outline clearly what the content should and should not contain.
3. Step-by-step instructions are provided for the LLM to grade the content.
4. The LLM returns a discrete score from 1-5.

### Setting guardrail thresholds

Our output guardrail will assess the LLM's response and block anything scoring a 3 or higher. Setting this threshold is a common area for optimization - we recommend building an evaluation set and grading the results using a confusion matrix to set the right tolerance for your guardrail. The trade-off here is generally:
- More false positives leads to a fractured user experience, where customers get annoyed and the assistant seems less helpful.
- More false negatives can cause lasting harm to your business, as people get the assistant to answer inappropriate questions, or prompt inject/jailbreak it.

For example, for jailbreaking you may want to have a very low threshold, as the risk to your business if your LLM is hijacked and used to produce dangerous content that ends up on social media is very high. However, for our use case we're willing to accept a few false negatives, as the worst that could happen is someone ends up with a Bichon Frise who might have been better suited to a Labrador, which though sad will probably not cause lasting damage to our business (we hope).


key: _node_content
string_value:
{"id_": "d5f166be-ef95-489d-8fe2-eea89578b8d0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1f04ad9-37a2-453e-bfa6-be4fca0c8808", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c58e86e1c1d1fd08d15d09e348385f9d3664a6f935c5ded82b1fc9e025d14305", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8341a892-6862-4d59-8bab-9bfe718fcfd7", "node_type": "1", "metadata": {}, "hash": "25c0f8fe37a8db4a0612673a2f03e06b745cdda81076f13d32462e7426f1bc2f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13034, "end_char_idx": 16059, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
domain = "animal breed recommendation"

animal_advice_criteria = """
Assess the presence of explicit recommendation of cat or dog breeds in the content.
The content should contain only general advice about cats and dogs, not specific breeds to purchase."""

animal_advice_steps = """
1. Read the content and the criteria carefully.
2. Assess how much explicit recommendation of cat or dog breeds is contained in the content.
3. Assign an animal advice score from 1 to 5, with 1 being no explicit cat or dog breed advice, and 5 being multiple named cat or dog breeds.
"""

moderation_system_prompt = """
You are a moderation assistant. Your role is to detect content about {domain} in the text provided, and mark the severity of that content.

## {domain}

### Criteria

{scoring_criteria}

### Instructions

{scoring_steps}

### Content

{content}

### Evaluation (score only!)
"""
```


```python
async def moderation_guardrail(chat_response):
    print("Checking moderation guardrail")
    mod_messages = [
        {"role": "user", "content": moderation_system_prompt.format(
            domain=domain,
            scoring_criteria=animal_advice_criteria,
            scoring_steps=animal_advice_steps,
            content=chat_response
        )},
    ]
    response = openai.chat.completions.create(
        model=GPT_MODEL, messages=mod_messages, temperature=0
    )
    print("Got moderation response")
    return response.choices[0].message.content
    
    
async def execute_all_guardrails(user_request):
    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))
    chat_task = asyncio.create_task(get_chat_response(user_request))

    while True:
        done, _ = await asyncio.wait(
            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED
        )
        if topical_guardrail_task in done:
            guardrail_response = topical_guardrail_task.result()
            if guardrail_response == "not_allowed":
                chat_task.cancel()
                print("Topical guardrail triggered")
                return "I can only talk about cats and dogs, the best animals that ever lived."
            elif chat_task in done:
                chat_response = chat_task.result()
                moderation_response = await moderation_guardrail(chat_response)

                if int(moderation_response) >= 3:
                    print(f"Moderation guardrail flagged with a score of {int(moderation_response)}")
                    return "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have."

                else:
                    print('Passed moderation')
                    return chat_response
        else:
            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again
```


```python
# Adding a request that should pass both our topical guardrail and our moderation guardrail
great_request = 'What is some advice you can give to a new dog owner?'
```


key: _node_content
string_value:
{"id_": "8341a892-6862-4d59-8bab-9bfe718fcfd7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ef4f73c1-4da8-49ea-a944-3f583aaa4ac3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "337c350bcca7d75854758eb0cad65acdf3e435a6ccea5bdebc39c067aeee844b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5f166be-ef95-489d-8fe2-eea89578b8d0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_use_guardrails.ipynb", "title": "How to use guardrails", "tokens": 3850, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "038b63a1ec3554db44f031cc88749560a894f30f38b3cfdab123e0a5a44b41c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15883, "end_char_idx": 18240, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Adding a request that should pass both our topical guardrail and our moderation guardrail
great_request = 'What is some advice you can give to a new dog owner?'
```


```python
tests = [good_request,bad_request,great_request]

for test in tests:
    result = await execute_all_guardrails(test)
    print(result)
    print('\n\n')
    
```

    Checking topical guardrail
    Got guardrail response
    Getting LLM response
    Got LLM response
    Checking moderation guardrail
    Got moderation response
    Moderation guardrail flagged with a score of 5
    Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.
    
    
    
    Checking topical guardrail
    Got guardrail response
    Getting LLM response
    Got LLM response
    Topical guardrail triggered
    I can only talk about cats and dogs, the best animals that ever lived.
    
    
    
    Checking topical guardrail
    Got guardrail response
    Getting LLM response
    Got LLM response
    Checking moderation guardrail
    Got moderation response
    Moderation guardrail flagged with a score of 3
    Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.
    
    
    


## Conclusion

Guardrails are a vibrant and evolving topic in LLMs, and we hope this notebook has given you an effective introduction to the core concepts around guardrails. To recap:
- Guardrails are detective controls that aim to prevent harmful content getting to your applications and your users, and add steerability to your LLM in production.
- They can take the form of input guardrails, which target content before it gets to the LLM, and output guardrails, which control the LLM's response.
- Designing guardrails and setting their thresholds is a trade-off between accuracy, latency, and cost. Your decision should be based on clear evaluations of the performance of your guardrails, and an understanding of what the cost of a false negative and false positive are for your business.
- By embracing asynchronous design principles, you can scale guardrails horizontally to minimize the impact to the user as your guardrails increase in number and scope.

We look forward to seeing how you take this forward, and how thinking on guardrails evolves as the ecosystem matures.


key: _node_content
string_value:
{"id_": "12b98990-475d-4d99-bf1f-8d5afebee4f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_embeddings.ipynb", "title": "Negative example (slow and rate-limited)", "tokens": 380, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "abbcbdc6-61b1-463e-96bd-789ec462d576", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Using_embeddings.ipynb", "title": "Negative example (slow and rate-limited)", "tokens": 380, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "26a585da0ade981944c372d689088efba5206abe28955e699674ffa1a4c727f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1581, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Using embeddings

This notebook contains some helpful snippets you can use to embed text with the `text-embedding-3-small` model via the OpenAI API.


```python
from openai import OpenAI
client = OpenAI()

embedding = client.embeddings.create(
    input="Your text goes here", model="text-embedding-3-small"
).data[0].embedding
len(embedding)

```




    1536



It's recommended to use the 'tenacity' package or another exponential backoff implementation to better manage API rate limits, as hitting the API too much too fast can trigger rate limits. Using the following function ensures you get your embeddings as fast as possible.


```python
# Negative example (slow and rate-limited)
from openai import OpenAI
client = OpenAI()

num_embeddings = 10000 # Some large number
for i in range(num_embeddings):
    embedding = client.embeddings.create(
        input="Your text goes here", model="text-embedding-3-small"
    ).data[0].embedding
    print(len(embedding))
```


```python
# Best practice
from tenacity import retry, wait_random_exponential, stop_after_attempt
from openai import OpenAI
client = OpenAI()

# Retry up to 6 times with exponential backoff, starting at 1 second and maxing out at 20 seconds delay
@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6))
def get_embedding(text: str, model="text-embedding-3-small") -> list[float]:
    return client.embeddings.create(input=[text], model=model).data[0].embedding

embedding = get_embedding("Your text goes here", model="text-embedding-3-small")
print(len(embedding))
```

    1536


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb


key: _node_content
string_value:
{"id_": "ca5002ca-4f63-4a4f-a0a8-fb914686daa7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0712c7bb-2778-4ab2-bd56-2b09d5074694", "node_type": "1", "metadata": {}, "hash": "2aac946d934a339e30acde13fdfe37b5d6578b90ead86bf76aaaae023c17087c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2956, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Embedding Wikipedia articles for search

This notebook shows how we prepared a dataset of Wikipedia articles for search, used in [Question_answering_using_embeddings.ipynb](Question_answering_using_embeddings.ipynb).

Procedure:

0. Prerequisites: Import libraries, set API key (if needed)
1. Collect: We download a few hundred Wikipedia articles about the 2022 Olympics
2. Chunk: Documents are split into short, semi-self-contained sections to be embedded
3. Embed: Each section is embedded with the OpenAI API
4. Store: Embeddings are saved in a CSV file (for large datasets, use a vector database)

## 0. Prerequisites

### Import libraries


```python
# imports
import mwclient  # for downloading example Wikipedia articles
import mwparserfromhell  # for splitting Wikipedia articles into sections
import openai  # for generating embeddings
import os  # for environment variables
import pandas as pd  # for DataFrames to store article sections and embeddings
import re  # for cutting <ref> links out of Wikipedia articles
import tiktoken  # for counting tokens

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

Install any missing libraries with `pip install` in your terminal. E.g.,

```zsh
pip install openai
```

(You can also do this in a notebook cell with `!pip install openai`.)

If you install any libraries, be sure to restart the notebook kernel.

### Set API key (if needed)

Note that the OpenAI library will try to read your API key from the `OPENAI_API_KEY` environment variable. If you haven't already, set this environment variable by following [these instructions](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).

## 1. Collect documents

In this example, we'll download a few hundred Wikipedia articles related to the 2022 Winter Olympics.


```python
# get Wikipedia pages about the 2022 Winter Olympics

CATEGORY_TITLE = "Category:2022 Winter Olympics"
WIKI_SITE = "en.wikipedia.org"


def titles_from_category(
    category: mwclient.listing.Category, max_depth: int
) -> set[str]:
    """Return a set of page titles in a given Wiki category and its subcategories."""
    titles = set()
    for cm in category.members():
        if type(cm) == mwclient.page.Page:
            # ^type() used instead of isinstance() to catch match w/ no inheritance
            titles.add(cm.name)
        elif isinstance(cm, mwclient.listing.Category) and max_depth > 0:
            deeper_titles = titles_from_category(cm, max_depth=max_depth - 1)
            titles.update(deeper_titles)
    return titles


site = mwclient.Site(WIKI_SITE)
category_page = site.pages[CATEGORY_TITLE]
titles = titles_from_category(category_page, max_depth=1)
# ^note: max_depth=1 means we go one level deep in the category tree
print(f"Found {len(titles)} article titles in {CATEGORY_TITLE}.")

```

    Found 731 article titles in Category:2022 Winter Olympics.


key: _node_content
string_value:
{"id_": "0712c7bb-2778-4ab2-bd56-2b09d5074694", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca5002ca-4f63-4a4f-a0a8-fb914686daa7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce84c7cd3733cb98d42d6a007ffcf0ec9e2402cbd5f4c47fc395a86abd6768a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e3ab0bd-03a5-4cd5-a0b3-f3c3fc7410e2", "node_type": "1", "metadata": {}, "hash": "534b7fa4171e7091d90093be9202722096521ebfbd5eb8bcd2f4ad43f2f76c74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1866, "end_char_idx": 4140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# get Wikipedia pages about the 2022 Winter Olympics

CATEGORY_TITLE = "Category:2022 Winter Olympics"
WIKI_SITE = "en.wikipedia.org"


def titles_from_category(
    category: mwclient.listing.Category, max_depth: int
) -> set[str]:
    """Return a set of page titles in a given Wiki category and its subcategories."""
    titles = set()
    for cm in category.members():
        if type(cm) == mwclient.page.Page:
            # ^type() used instead of isinstance() to catch match w/ no inheritance
            titles.add(cm.name)
        elif isinstance(cm, mwclient.listing.Category) and max_depth > 0:
            deeper_titles = titles_from_category(cm, max_depth=max_depth - 1)
            titles.update(deeper_titles)
    return titles


site = mwclient.Site(WIKI_SITE)
category_page = site.pages[CATEGORY_TITLE]
titles = titles_from_category(category_page, max_depth=1)
# ^note: max_depth=1 means we go one level deep in the category tree
print(f"Found {len(titles)} article titles in {CATEGORY_TITLE}.")

```

    Found 731 article titles in Category:2022 Winter Olympics.


## 2. Chunk documents

Now that we have our reference documents, we need to prepare them for search.

Because GPT can only read a limited amount of text at once, we'll split each document into chunks short enough to be read.

For this specific example on Wikipedia articles, we'll:
- Discard less relevant-looking sections like External Links and Footnotes
- Clean up the text by removing reference tags (e.g., <ref>), whitespace, and super short sections
- Split each article into sections
- Prepend titles and subtitles to each section's text, to help GPT understand the context
- If a section is long (say, > 1,600 tokens), we'll recursively split it into smaller sections, trying to split along semantic boundaries like paragraphs


```python
# define functions to split Wikipedia pages into sections

SECTIONS_TO_IGNORE = [
    "See also",
    "References",
    "External links",
    "Further reading",
    "Footnotes",
    "Bibliography",
    "Sources",
    "Citations",
    "Literature",
    "Footnotes",
    "Notes and references",
    "Photo gallery",
    "Works cited",
    "Photos",
    "Gallery",
    "Notes",
    "References and sources",
    "References and notes",
]


key: _node_content
string_value:
{"id_": "8e3ab0bd-03a5-4cd5-a0b3-f3c3fc7410e2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0712c7bb-2778-4ab2-bd56-2b09d5074694", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "de3030c1d12607e6c88c0e201950a391e154c2aa45c887e19e5b0e1abc1bac4e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7baf7634-d75c-4287-9771-6294f423d5fd", "node_type": "1", "metadata": {}, "hash": "ab6de8e84580e2e728db30e92221ead9e96c6ed9b28725d473f06200dbe7e0a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2620, "end_char_idx": 5364, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
site = mwclient.Site(WIKI_SITE)
category_page = site.pages[CATEGORY_TITLE]
titles = titles_from_category(category_page, max_depth=1)
# ^note: max_depth=1 means we go one level deep in the category tree
print(f"Found {len(titles)} article titles in {CATEGORY_TITLE}.")

```

    Found 731 article titles in Category:2022 Winter Olympics.


## 2. Chunk documents

Now that we have our reference documents, we need to prepare them for search.

Because GPT can only read a limited amount of text at once, we'll split each document into chunks short enough to be read.

For this specific example on Wikipedia articles, we'll:
- Discard less relevant-looking sections like External Links and Footnotes
- Clean up the text by removing reference tags (e.g., <ref>), whitespace, and super short sections
- Split each article into sections
- Prepend titles and subtitles to each section's text, to help GPT understand the context
- If a section is long (say, > 1,600 tokens), we'll recursively split it into smaller sections, trying to split along semantic boundaries like paragraphs


```python
# define functions to split Wikipedia pages into sections

SECTIONS_TO_IGNORE = [
    "See also",
    "References",
    "External links",
    "Further reading",
    "Footnotes",
    "Bibliography",
    "Sources",
    "Citations",
    "Literature",
    "Footnotes",
    "Notes and references",
    "Photo gallery",
    "Works cited",
    "Photos",
    "Gallery",
    "Notes",
    "References and sources",
    "References and notes",
]


def all_subsections_from_section(
    section: mwparserfromhell.wikicode.Wikicode,
    parent_titles: list[str],
    sections_to_ignore: set[str],
) -> list[tuple[list[str], str]]:
    """
    From a Wikipedia section, return a flattened list of all nested subsections.
    Each subsection is a tuple, where:
        - the first element is a list of parent subtitles, starting with the page title
        - the second element is the text of the subsection (but not any children)
    """
    headings = [str(h) for h in section.filter_headings()]
    title = headings[0]
    if title.strip("=" + " ") in sections_to_ignore:
        # ^wiki headings are wrapped like "== Heading =="
        return []
    titles = parent_titles + [title]
    full_text = str(section)
    section_text = full_text.split(title)[1]
    if len(headings) == 1:
        return [(titles, section_text)]
    else:
        first_subtitle = headings[1]
        section_text = section_text.split(first_subtitle)[0]
        results = [(titles, section_text)]
        for subsection in section.get_sections(levels=[len(titles) + 1]):
            results.extend(all_subsections_from_section(subsection, titles, sections_to_ignore))
        return results


key: _node_content
string_value:
{"id_": "7baf7634-d75c-4287-9771-6294f423d5fd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e3ab0bd-03a5-4cd5-a0b3-f3c3fc7410e2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a4b36fa230c39382991f6e51a78627dad58180648c515750fd9508705c23edb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52ddc887-4014-49ce-93df-36c33e8f0049", "node_type": "1", "metadata": {}, "hash": "fef1b0557444b23cb6c3158919f1093d088e727cbf6883239f7dfb73d78a5f9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3696, "end_char_idx": 6691, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# define functions to split Wikipedia pages into sections

SECTIONS_TO_IGNORE = [
    "See also",
    "References",
    "External links",
    "Further reading",
    "Footnotes",
    "Bibliography",
    "Sources",
    "Citations",
    "Literature",
    "Footnotes",
    "Notes and references",
    "Photo gallery",
    "Works cited",
    "Photos",
    "Gallery",
    "Notes",
    "References and sources",
    "References and notes",
]


def all_subsections_from_section(
    section: mwparserfromhell.wikicode.Wikicode,
    parent_titles: list[str],
    sections_to_ignore: set[str],
) -> list[tuple[list[str], str]]:
    """
    From a Wikipedia section, return a flattened list of all nested subsections.
    Each subsection is a tuple, where:
        - the first element is a list of parent subtitles, starting with the page title
        - the second element is the text of the subsection (but not any children)
    """
    headings = [str(h) for h in section.filter_headings()]
    title = headings[0]
    if title.strip("=" + " ") in sections_to_ignore:
        # ^wiki headings are wrapped like "== Heading =="
        return []
    titles = parent_titles + [title]
    full_text = str(section)
    section_text = full_text.split(title)[1]
    if len(headings) == 1:
        return [(titles, section_text)]
    else:
        first_subtitle = headings[1]
        section_text = section_text.split(first_subtitle)[0]
        results = [(titles, section_text)]
        for subsection in section.get_sections(levels=[len(titles) + 1]):
            results.extend(all_subsections_from_section(subsection, titles, sections_to_ignore))
        return results


def all_subsections_from_title(
    title: str,
    sections_to_ignore: set[str] = SECTIONS_TO_IGNORE,
    site_name: str = WIKI_SITE,
) -> list[tuple[list[str], str]]:
    """From a Wikipedia page title, return a flattened list of all nested subsections.
    Each subsection is a tuple, where:
        - the first element is a list of parent subtitles, starting with the page title
        - the second element is the text of the subsection (but not any children)
    """
    site = mwclient.Site(site_name)
    page = site.pages[title]
    text = page.text()
    parsed_text = mwparserfromhell.parse(text)
    headings = [str(h) for h in parsed_text.filter_headings()]
    if headings:
        summary_text = str(parsed_text).split(headings[0])[0]
    else:
        summary_text = str(parsed_text)
    results = [([title], summary_text)]
    for subsection in parsed_text.get_sections(levels=[2]):
        results.extend(all_subsections_from_section(subsection, [title], sections_to_ignore))
    return results

```


```python
# split pages into sections
# may take ~1 minute per 100 articles
wikipedia_sections = []
for title in titles:
    wikipedia_sections.extend(all_subsections_from_title(title))
print(f"Found {len(wikipedia_sections)} sections in {len(titles)} pages.")

```

    Found 5730 sections in 731 pages.


key: _node_content
string_value:
{"id_": "52ddc887-4014-49ce-93df-36c33e8f0049", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7baf7634-d75c-4287-9771-6294f423d5fd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b19edb443b084737d88172bddf8c2ab4fa055dcf710601b471ba2300e6f4bd52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95dcf96d-ab5d-4ae3-844a-04ab4cc13779", "node_type": "1", "metadata": {}, "hash": "823da9c5677e327c9be9e55a9f0fc7e220f97c0062c24d17da43f28ff7bff32f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5367, "end_char_idx": 8384, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def all_subsections_from_title(
    title: str,
    sections_to_ignore: set[str] = SECTIONS_TO_IGNORE,
    site_name: str = WIKI_SITE,
) -> list[tuple[list[str], str]]:
    """From a Wikipedia page title, return a flattened list of all nested subsections.
    Each subsection is a tuple, where:
        - the first element is a list of parent subtitles, starting with the page title
        - the second element is the text of the subsection (but not any children)
    """
    site = mwclient.Site(site_name)
    page = site.pages[title]
    text = page.text()
    parsed_text = mwparserfromhell.parse(text)
    headings = [str(h) for h in parsed_text.filter_headings()]
    if headings:
        summary_text = str(parsed_text).split(headings[0])[0]
    else:
        summary_text = str(parsed_text)
    results = [([title], summary_text)]
    for subsection in parsed_text.get_sections(levels=[2]):
        results.extend(all_subsections_from_section(subsection, [title], sections_to_ignore))
    return results

```


```python
# split pages into sections
# may take ~1 minute per 100 articles
wikipedia_sections = []
for title in titles:
    wikipedia_sections.extend(all_subsections_from_title(title))
print(f"Found {len(wikipedia_sections)} sections in {len(titles)} pages.")

```

    Found 5730 sections in 731 pages.



```python
# clean text
def clean_section(section: tuple[list[str], str]) -> tuple[list[str], str]:
    """
    Return a cleaned up section with:
        - <ref>xyz</ref> patterns removed
        - leading/trailing whitespace removed
    """
    titles, text = section
    text = re.sub(r"<ref.*?</ref>", "", text)
    text = text.strip()
    return (titles, text)


wikipedia_sections = [clean_section(ws) for ws in wikipedia_sections]

# filter out short/blank sections
def keep_section(section: tuple[list[str], str]) -> bool:
    """Return True if the section should be kept, False otherwise."""
    titles, text = section
    if len(text) < 16:
        return False
    else:
        return True


original_num_sections = len(wikipedia_sections)
wikipedia_sections = [ws for ws in wikipedia_sections if keep_section(ws)]
print(f"Filtered out {original_num_sections-len(wikipedia_sections)} sections, leaving {len(wikipedia_sections)} sections.")

```

    Filtered out 530 sections, leaving 5200 sections.



```python
# print example data
for ws in wikipedia_sections[:5]:
    print(ws[0])
    display(ws[1][:77] + "...")
    print()

```

    ['Lviv bid for the 2022 Winter Olympics']



    '{{Olympic bid|2022|Winter|\n| Paralympics = yes\n| logo = Lviv 2022 Winter Olym...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==History==']



    '[[Image:Lwów - Rynek 01.JPG|thumb|right|200px|View of Rynok Square in Lviv]]\n...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==']



    '{{Location map+\n|Ukraine\n|border =\n|caption = Venue areas\n|float = left\n|widt...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==', '===City zone===']


key: _node_content
string_value:
{"id_": "95dcf96d-ab5d-4ae3-844a-04ab4cc13779", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52ddc887-4014-49ce-93df-36c33e8f0049", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7df6de4339ce22b4cac3964bff84bf402cf21356b3f9fd28c8a5bec7e38da951", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60cb65db-860e-44a3-bb8a-c57c6630a449", "node_type": "1", "metadata": {}, "hash": "54965f447fbdbf6114035709ce5edac391c4994da6e4253b4960c3d23448b7ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7061, "end_char_idx": 9824, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
wikipedia_sections = [clean_section(ws) for ws in wikipedia_sections]

# filter out short/blank sections
def keep_section(section: tuple[list[str], str]) -> bool:
    """Return True if the section should be kept, False otherwise."""
    titles, text = section
    if len(text) < 16:
        return False
    else:
        return True


original_num_sections = len(wikipedia_sections)
wikipedia_sections = [ws for ws in wikipedia_sections if keep_section(ws)]
print(f"Filtered out {original_num_sections-len(wikipedia_sections)} sections, leaving {len(wikipedia_sections)} sections.")

```

    Filtered out 530 sections, leaving 5200 sections.



```python
# print example data
for ws in wikipedia_sections[:5]:
    print(ws[0])
    display(ws[1][:77] + "...")
    print()

```

    ['Lviv bid for the 2022 Winter Olympics']



    '{{Olympic bid|2022|Winter|\n| Paralympics = yes\n| logo = Lviv 2022 Winter Olym...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==History==']



    '[[Image:Lwów - Rynek 01.JPG|thumb|right|200px|View of Rynok Square in Lviv]]\n...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==']



    '{{Location map+\n|Ukraine\n|border =\n|caption = Venue areas\n|float = left\n|widt...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==', '===City zone===']



    'The main Olympic Park would be centered around the [[Arena Lviv]], hosting th...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==', '===Mountain zone===', '====Venue cluster Tysovets-Panasivka====']



    'An existing military ski training facility in [[Tysovets, Skole Raion|Tysovet...'


    


Next, we'll recursively split long sections into smaller sections.

There's no perfect recipe for splitting text into sections.

Some tradeoffs include:
- Longer sections may be better for questions that require more context
- Longer sections may be worse for retrieval, as they may have more topics muddled together
- Shorter sections are better for reducing costs (which are proportional to the number of tokens)
- Shorter sections allow more sections to be retrieved, which may help with recall
- Overlapping sections may help prevent answers from being cut by section boundaries

Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible.


```python
GPT_MODEL = "gpt-3.5-turbo"  # only matters insofar as it selects which tokenizer to use


def num_tokens(text: str, model: str = GPT_MODEL) -> int:
    """Return the number of tokens in a string."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


key: _node_content
string_value:
{"id_": "60cb65db-860e-44a3-bb8a-c57c6630a449", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95dcf96d-ab5d-4ae3-844a-04ab4cc13779", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "72188b5fecef06d3807bc35031bd0b60d6b47ea4cd3b0e7d96ac4fbf18e6793b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d48f6f4-4ced-4973-bd2e-0964b9aa4609", "node_type": "1", "metadata": {}, "hash": "9262974da17f999f2b938a2262e342d664fbc8a3f904ca70d63b3cc816ab6c41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8211, "end_char_idx": 11244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'{{Location map+\n|Ukraine\n|border =\n|caption = Venue areas\n|float = left\n|widt...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==', '===City zone===']



    'The main Olympic Park would be centered around the [[Arena Lviv]], hosting th...'


    
    ['Lviv bid for the 2022 Winter Olympics', '==Venues==', '===Mountain zone===', '====Venue cluster Tysovets-Panasivka====']



    'An existing military ski training facility in [[Tysovets, Skole Raion|Tysovet...'


    


Next, we'll recursively split long sections into smaller sections.

There's no perfect recipe for splitting text into sections.

Some tradeoffs include:
- Longer sections may be better for questions that require more context
- Longer sections may be worse for retrieval, as they may have more topics muddled together
- Shorter sections are better for reducing costs (which are proportional to the number of tokens)
- Shorter sections allow more sections to be retrieved, which may help with recall
- Overlapping sections may help prevent answers from being cut by section boundaries

Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible.


```python
GPT_MODEL = "gpt-3.5-turbo"  # only matters insofar as it selects which tokenizer to use


def num_tokens(text: str, model: str = GPT_MODEL) -> int:
    """Return the number of tokens in a string."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


def halved_by_delimiter(string: str, delimiter: str = "\n") -> list[str, str]:
    """Split a string in two, on a delimiter, trying to balance tokens on each side."""
    chunks = string.split(delimiter)
    if len(chunks) == 1:
        return [string, ""]  # no delimiter found
    elif len(chunks) == 2:
        return chunks  # no need to search for halfway point
    else:
        total_tokens = num_tokens(string)
        halfway = total_tokens // 2
        best_diff = halfway
        for i, chunk in enumerate(chunks):
            left = delimiter.join(chunks[: i + 1])
            left_tokens = num_tokens(left)
            diff = abs(halfway - left_tokens)
            if diff >= best_diff:
                break
            else:
                best_diff = diff
        left = delimiter.join(chunks[:i])
        right = delimiter.join(chunks[i:])
        return [left, right]


def truncated_string(
    string: str,
    model: str,
    max_tokens: int,
    print_warning: bool = True,
) -> str:
    """Truncate a string to a maximum number of tokens."""
    encoding = tiktoken.encoding_for_model(model)
    encoded_string = encoding.encode(string)
    truncated_string = encoding.decode(encoded_string[:max_tokens])
    if print_warning and len(encoded_string) > max_tokens:
        print(f"Warning: Truncated string from {len(encoded_string)} tokens to {max_tokens} tokens.")
    return truncated_string


key: _node_content
string_value:
{"id_": "4d48f6f4-4ced-4973-bd2e-0964b9aa4609", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60cb65db-860e-44a3-bb8a-c57c6630a449", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d3f9cbb9b9450377c6411a8c76ab4fb8eb20541fe70625b0aee467b0ced2d204", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7a0e512-c4a3-4dfd-b113-9052e34f96ac", "node_type": "1", "metadata": {}, "hash": "d0600a2d7fadbeb388d13462889003f391808220cbccc08090e5ee7ced8e6cc6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9628, "end_char_idx": 13049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def num_tokens(text: str, model: str = GPT_MODEL) -> int:
    """Return the number of tokens in a string."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


def halved_by_delimiter(string: str, delimiter: str = "\n") -> list[str, str]:
    """Split a string in two, on a delimiter, trying to balance tokens on each side."""
    chunks = string.split(delimiter)
    if len(chunks) == 1:
        return [string, ""]  # no delimiter found
    elif len(chunks) == 2:
        return chunks  # no need to search for halfway point
    else:
        total_tokens = num_tokens(string)
        halfway = total_tokens // 2
        best_diff = halfway
        for i, chunk in enumerate(chunks):
            left = delimiter.join(chunks[: i + 1])
            left_tokens = num_tokens(left)
            diff = abs(halfway - left_tokens)
            if diff >= best_diff:
                break
            else:
                best_diff = diff
        left = delimiter.join(chunks[:i])
        right = delimiter.join(chunks[i:])
        return [left, right]


def truncated_string(
    string: str,
    model: str,
    max_tokens: int,
    print_warning: bool = True,
) -> str:
    """Truncate a string to a maximum number of tokens."""
    encoding = tiktoken.encoding_for_model(model)
    encoded_string = encoding.encode(string)
    truncated_string = encoding.decode(encoded_string[:max_tokens])
    if print_warning and len(encoded_string) > max_tokens:
        print(f"Warning: Truncated string from {len(encoded_string)} tokens to {max_tokens} tokens.")
    return truncated_string


def split_strings_from_subsection(
    subsection: tuple[list[str], str],
    max_tokens: int = 1000,
    model: str = GPT_MODEL,
    max_recursion: int = 5,
) -> list[str]:
    """
    Split a subsection into a list of subsections, each with no more than max_tokens.
    Each subsection is a tuple of parent titles [H1, H2, ...] and text (str).
    """
    titles, text = subsection
    string = "\n\n".join(titles + [text])
    num_tokens_in_string = num_tokens(string)
    # if length is fine, return string
    if num_tokens_in_string <= max_tokens:
        return [string]
    # if recursion hasn't found a split after X iterations, just truncate
    elif max_recursion == 0:
        return [truncated_string(string, model=model, max_tokens=max_tokens)]
    # otherwise, split in half and recurse
    else:
        titles, text = subsection
        for delimiter in ["\n\n", "\n", ". "]:
            left, right = halved_by_delimiter(text, delimiter=delimiter)
            if left == "" or right == "":
                # if either half is empty, retry with a more fine-grained delimiter
                continue
            else:
                # recurse on each half
                results = []
                for half in [left, right]:
                    half_subsection = (titles, half)
                    half_strings = split_strings_from_subsection(
                        half_subsection,
                        max_tokens=max_tokens,
                        model=model,
                        max_recursion=max_recursion - 1,
                    )
                    results.extend(half_strings)
                return results
    # otherwise no split was found, so just truncate (should be very rare)
    return [truncated_string(string, model=model, max_tokens=max_tokens)]

```


key: _node_content
string_value:
{"id_": "a7a0e512-c4a3-4dfd-b113-9052e34f96ac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d48f6f4-4ced-4973-bd2e-0964b9aa4609", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d7a1258bd016f6c0d9393b3b19896fa9d9cfebc6dd7188bbd59b19cc14ba5e00", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e822c197-d090-4cab-900e-5a17f2949ef8", "node_type": "1", "metadata": {}, "hash": "03a3f10f5abeb9397fa17a2edcedc1a05eed3edeaadde150b4c31f3e2fbd6173", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11247, "end_char_idx": 13418, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def split_strings_from_subsection(
    subsection: tuple[list[str], str],
    max_tokens: int = 1000,
    model: str = GPT_MODEL,
    max_recursion: int = 5,
) -> list[str]:
    """
    Split a subsection into a list of subsections, each with no more than max_tokens.
    Each subsection is a tuple of parent titles [H1, H2, ...] and text (str).
    """
    titles, text = subsection
    string = "\n\n".join(titles + [text])
    num_tokens_in_string = num_tokens(string)
    # if length is fine, return string
    if num_tokens_in_string <= max_tokens:
        return [string]
    # if recursion hasn't found a split after X iterations, just truncate
    elif max_recursion == 0:
        return [truncated_string(string, model=model, max_tokens=max_tokens)]
    # otherwise, split in half and recurse
    else:
        titles, text = subsection
        for delimiter in ["\n\n", "\n", ". "]:
            left, right = halved_by_delimiter(text, delimiter=delimiter)
            if left == "" or right == "":
                # if either half is empty, retry with a more fine-grained delimiter
                continue
            else:
                # recurse on each half
                results = []
                for half in [left, right]:
                    half_subsection = (titles, half)
                    half_strings = split_strings_from_subsection(
                        half_subsection,
                        max_tokens=max_tokens,
                        model=model,
                        max_recursion=max_recursion - 1,
                    )
                    results.extend(half_strings)
                return results
    # otherwise no split was found, so just truncate (should be very rare)
    return [truncated_string(string, model=model, max_tokens=max_tokens)]

```


```python
# split sections into chunks
MAX_TOKENS = 1600
wikipedia_strings = []
for section in wikipedia_sections:
    wikipedia_strings.extend(split_strings_from_subsection(section, max_tokens=MAX_TOKENS))

print(f"{len(wikipedia_sections)} Wikipedia sections split into {len(wikipedia_strings)} strings.")

```

    5200 Wikipedia sections split into 6059 strings.


key: _node_content
string_value:
{"id_": "e822c197-d090-4cab-900e-5a17f2949ef8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a7a0e512-c4a3-4dfd-b113-9052e34f96ac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bb76f19be42be0c32e510658253236da270b2ce6fdefa6562ff37775a295e00f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fdca2244-6a12-4c50-abf1-76f3cbb04a39", "node_type": "1", "metadata": {}, "hash": "718d3a26e40cb050b114ec7bfa6bc261ad029897cdb8a6345beb434b43436e3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13052, "end_char_idx": 16232, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# split sections into chunks
MAX_TOKENS = 1600
wikipedia_strings = []
for section in wikipedia_sections:
    wikipedia_strings.extend(split_strings_from_subsection(section, max_tokens=MAX_TOKENS))

print(f"{len(wikipedia_sections)} Wikipedia sections split into {len(wikipedia_strings)} strings.")

```

    5200 Wikipedia sections split into 6059 strings.



```python
# print example data
print(wikipedia_strings[1])

```

    Lviv bid for the 2022 Winter Olympics
    
    ==History==
    
    [[Image:Lwów - Rynek 01.JPG|thumb|right|200px|View of Rynok Square in Lviv]]
    
    On 27 May 2010, [[President of Ukraine]] [[Viktor Yanukovych]] stated during a visit to [[Lviv]] that Ukraine "will start working on the official nomination of our country as the holder of the Winter Olympic Games in [[Carpathian Mountains|Carpathians]]".
    
    In September 2012, [[government of Ukraine]] approved a document about the technical-economic substantiation of the national project "Olympic Hope 2022". This was announced by Vladyslav Kaskiv, the head of Ukraine´s Derzhinvestproekt (State investment project). The organizers announced on their website venue plans featuring Lviv as the host city and location for the "ice sport" venues, [[Volovets]] (around {{convert|185|km|mi|abbr=on}} from Lviv) as venue for the [[Alpine skiing]] competitions and [[Tysovets, Skole Raion|Tysovets]] (around {{convert|130|km|mi|abbr=on}} from Lviv) as venue for all other "snow sport" competitions. By March 2013 no other preparations than the feasibility study had been approved.
    
    On 24 October 2013, session of the Lviv City Council adopted a resolution "About submission to the International Olympic Committee for nomination of city to participate in the procedure for determining the host city of Olympic and Paralympic Winter Games in 2022".
    
    On 5 November 2013, it was confirmed that Lviv was bidding to host the [[2022 Winter Olympics]]. Lviv would host the ice sport events, while the skiing events would be held in the [[Carpathian]] mountains. This was the first bid Ukraine had ever submitted for an Olympic Games.
    
    On 30 June 2014, the International Olympic Committee announced "Lviv will turn its attention to an Olympic bid for 2026, and not continue with its application for 2022. The decision comes as a result of the present political and economic circumstances in Ukraine."
    
    Ukraine's Deputy Prime Minister Oleksandr Vilkul said that the Winter Games "will be an impetus not just for promotion of sports and tourism in Ukraine, but a very important component in the economic development of Ukraine, the attraction of the investments, the creation of new jobs, opening Ukraine to the world, returning Ukrainians working abroad to their motherland."
    
    Lviv was one of the host cities of [[UEFA Euro 2012]].


## 3. Embed document chunks

Now that we've split our library into shorter self-contained strings, we can compute embeddings for each.

(For large embedding jobs, use a script like [api_request_parallel_processor.py](api_request_parallel_processor.py) to parallelize requests while throttling to stay under rate limits.)


key: _node_content
string_value:
{"id_": "fdca2244-6a12-4c50-abf1-76f3cbb04a39", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c5ad8cb5-e379-48c2-a88f-1f88d025f030", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8215343c9233df1c01dfdbaeefe91e68909b503c1cc90e947c8a8f0e951b2f47", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e822c197-d090-4cab-900e-5a17f2949ef8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_Wikipedia_articles_for_search.ipynb", "title": "Embedding Wikipedia articles for search", "tokens": 4092, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0650e99c38d7c4b15e9105319e5611dc1e45f9cf1a0d3ce6d7090476b2d62cd3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15912, "end_char_idx": 17474, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 3. Embed document chunks

Now that we've split our library into shorter self-contained strings, we can compute embeddings for each.

(For large embedding jobs, use a script like [api_request_parallel_processor.py](api_request_parallel_processor.py) to parallelize requests while throttling to stay under rate limits.)


```python
EMBEDDING_MODEL = "text-embedding-3-small"
BATCH_SIZE = 1000  # you can submit up to 2048 embedding inputs per request

embeddings = []
for batch_start in range(0, len(wikipedia_strings), BATCH_SIZE):
    batch_end = batch_start + BATCH_SIZE
    batch = wikipedia_strings[batch_start:batch_end]
    print(f"Batch {batch_start} to {batch_end-1}")
    response = client.embeddings.create(model=EMBEDDING_MODEL, input=batch)
    for i, be in enumerate(response.data):
        assert i == be.index  # double check embeddings are in same order as input
    batch_embeddings = [e.embedding for e in response.data]
    embeddings.extend(batch_embeddings)

df = pd.DataFrame({"text": wikipedia_strings, "embedding": embeddings})

```

    Batch 0 to 999
    Batch 1000 to 1999
    Batch 2000 to 2999
    Batch 3000 to 3999
    Batch 4000 to 4999
    Batch 5000 to 5999
    Batch 6000 to 6999


## 4. Store document chunks and embeddings

Because this example only uses a few thousand strings, we'll store them in a CSV file.

(For larger datasets, use a vector database, which will be more performant.)


```python
# save document chunks and embeddings

SAVE_PATH = "data/winter_olympics_2022.csv"

df.to_csv(SAVE_PATH, index=False)

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb


key: _node_content
string_value:
{"id_": "3b1b355a-caa7-4ec2-92ad-4bf3ee29cff0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96ae7cc6-598c-4f9a-bbd1-f1e9138d4841", "node_type": "1", "metadata": {}, "hash": "4dec753a6ccb3ac73b93413931f424fd9c2cf3644c9c8f1239090da0848e7f16", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to format inputs to ChatGPT models

ChatGPT is powered by `gpt-3.5-turbo` and `gpt-4`, OpenAI's most advanced models.

You can build your own applications with `gpt-3.5-turbo` or `gpt-4` using the OpenAI API.

Chat models take a series of messages as input, and return an AI-written message as output.

This guide illustrates the chat format with a few example API calls.

## 1. Import the openai library


```python
# if needed, install and/or upgrade to the latest version of the OpenAI Python library
%pip install --upgrade openai

```


```python
# import the OpenAI Python library for calling the OpenAI API
from openai import OpenAI
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

## 2. An example chat completion API call

A chat completion API call parameters,
**Required**
- `model`: the name of the model you want to use (e.g., `gpt-3.5-turbo`, `gpt-4`, `gpt-3.5-turbo-16k-1106`)
- `messages`: a list of message objects, where each object has two required fields:
    - `role`: the role of the messenger (either `system`, `user`, `assistant` or `tool`)
    - `content`: the content of the message (e.g., `Write me a beautiful poem`)

Messages can also contain an optional `name` field, which give the messenger a name. E.g., `example-user`, `Alice`, `BlackbeardBot`. Names may not contain spaces.

**Optional**
- `frequency_penalty`: Penalizes tokens based on their frequency, reducing repetition.
- `logit_bias`: Modifies likelihood of specified tokens with bias values.
- `logprobs`: Returns log probabilities of output tokens if true.
- `top_logprobs`: Specifies the number of most likely tokens to return at each position.
- `max_tokens`: Sets the maximum number of generated tokens in chat completion.
- `n`: Generates a specified number of chat completion choices for each input.
- `presence_penalty`: Penalizes new tokens based on their presence in the text.
- `response_format`: Specifies the output format, e.g., JSON mode.
- `seed`: Ensures deterministic sampling with a specified seed.
- `stop`: Specifies up to 4 sequences where the API should stop generating tokens.
- `stream`: Sends partial message deltas as tokens become available.
- `temperature`: Sets the sampling temperature between 0 and 2.
- `top_p`: Uses nucleus sampling; considers tokens with top_p probability mass.
- `tools`: Lists functions the model may call.
- `tool_choice`: Controls the model's function calls (none/auto/function).
- `user`: Unique identifier for end-user monitoring and abuse detection.


key: _node_content
string_value:
{"id_": "96ae7cc6-598c-4f9a-bbd1-f1e9138d4841", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b1b355a-caa7-4ec2-92ad-4bf3ee29cff0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a862f3dfa6b6a0cb3a26aef9d2881fd5be3752f25ccf8f06d0e92f50a62b0936", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35c97a9d-12fe-4cd3-8703-457c12d3dbaf", "node_type": "1", "metadata": {}, "hash": "7bf6863c6b5a4e1149307801a62b66f77b4f1e8ea9c3f578690c69e14759df3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2579, "end_char_idx": 5683, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As of January 2024, you can also optionally submit a list of `functions` that tell GPT whether it can generate JSON to feed into a function. For details, see the [documentation](https://platform.openai.com/docs/guides/function-calling), [API reference](https://platform.openai.com/docs/api-reference/chat), or the Cookbook guide [How to call functions with chat models](How_to_call_functions_with_chat_models.ipynb).

Typically, a conversation will start with a system message that tells the assistant how to behave, followed by alternating user and assistant messages, but you are not required to follow this format.

Let's look at an example chat API calls to see how the chat format works in practice.


```python
# Example OpenAI Python library request
MODEL = "gpt-3.5-turbo"
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Knock knock."},
        {"role": "assistant", "content": "Who's there?"},
        {"role": "user", "content": "Orange."},
    ],
    temperature=0,
)

```


```python
print(json.dumps(json.loads(response.model_dump_json()), indent=4))
```

    {
        "id": "chatcmpl-8dee9DuEFcg2QILtT2a6EBXZnpirM",
        "choices": [
            {
                "finish_reason": "stop",
                "index": 0,
                "logprobs": null,
                "message": {
                    "content": "Orange who?",
                    "role": "assistant",
                    "function_call": null,
                    "tool_calls": null
                }
            }
        ],
        "created": 1704461729,
        "model": "gpt-3.5-turbo-0613",
        "object": "chat.completion",
        "system_fingerprint": null,
        "usage": {
            "completion_tokens": 3,
            "prompt_tokens": 35,
            "total_tokens": 38
        }
    }


As you can see, the response object has a few fields:
- `id`: the ID of the request
- `choices`: a list of completion objects (only one, unless you set `n` greater than 1)
    - `finish_reason`: the reason the model stopped generating text (either `stop`, or `length` if `max_tokens` limit was reached)
    - `index`: The index of the choice in the list of choices.
    - `logprobs`: Log probability information for the choice.
    - `message`: the message object generated by the model
        - `content`: content of message
        - `role`: The role of the author of this message.
        - `tool_calls`: The tool calls generated by the model, such as function calls. if the tools is given
- `created`: the timestamp of the request
- `model`: the full name of the model used to generate the response
- `object`: the type of object returned (e.g., `chat.completion`)
- `system_fingerprint`: This fingerprint represents the backend configuration that the model runs with.
- `usage`: the number of tokens used to generate the replies, counting prompt, completion, and total

Extract just the reply with:


```python
response.choices[0].message.content

```




    'Orange who?'


key: _node_content
string_value:
{"id_": "35c97a9d-12fe-4cd3-8703-457c12d3dbaf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96ae7cc6-598c-4f9a-bbd1-f1e9138d4841", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2015b98716ca258c45a9301f33f760d1378e6f5a04756f4646d97982b6e50e78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56b9e192-f132-44c6-b884-6fbcf4552789", "node_type": "1", "metadata": {}, "hash": "be5ed72e0863a08a06d0636c7468a80fa120f48c410be10dd035d21a952e5a1c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4504, "end_char_idx": 5952, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As you can see, the response object has a few fields:
- `id`: the ID of the request
- `choices`: a list of completion objects (only one, unless you set `n` greater than 1)
    - `finish_reason`: the reason the model stopped generating text (either `stop`, or `length` if `max_tokens` limit was reached)
    - `index`: The index of the choice in the list of choices.
    - `logprobs`: Log probability information for the choice.
    - `message`: the message object generated by the model
        - `content`: content of message
        - `role`: The role of the author of this message.
        - `tool_calls`: The tool calls generated by the model, such as function calls. if the tools is given
- `created`: the timestamp of the request
- `model`: the full name of the model used to generate the response
- `object`: the type of object returned (e.g., `chat.completion`)
- `system_fingerprint`: This fingerprint represents the backend configuration that the model runs with.
- `usage`: the number of tokens used to generate the replies, counting prompt, completion, and total

Extract just the reply with:


```python
response.choices[0].message.content

```




    'Orange who?'



Even non-conversation-based tasks can fit into the chat format, by placing the instruction in the first user message.

For example, to ask the model to explain asynchronous programming in the style of the pirate Blackbeard, we can structure conversation as follows:


key: _node_content
string_value:
{"id_": "56b9e192-f132-44c6-b884-6fbcf4552789", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35c97a9d-12fe-4cd3-8703-457c12d3dbaf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a732caaac850ea12746870547a0dce795a4ba17c17dd2bc7cb9944ed18f8c15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f61fa2ba-d8a0-4210-b298-7f4e19dbf4c7", "node_type": "1", "metadata": {}, "hash": "7ba9b13400d2056690a65416b53e468599581932ac4ee8d9a98089eab32ec03f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4504, "end_char_idx": 8544, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As you can see, the response object has a few fields:
- `id`: the ID of the request
- `choices`: a list of completion objects (only one, unless you set `n` greater than 1)
    - `finish_reason`: the reason the model stopped generating text (either `stop`, or `length` if `max_tokens` limit was reached)
    - `index`: The index of the choice in the list of choices.
    - `logprobs`: Log probability information for the choice.
    - `message`: the message object generated by the model
        - `content`: content of message
        - `role`: The role of the author of this message.
        - `tool_calls`: The tool calls generated by the model, such as function calls. if the tools is given
- `created`: the timestamp of the request
- `model`: the full name of the model used to generate the response
- `object`: the type of object returned (e.g., `chat.completion`)
- `system_fingerprint`: This fingerprint represents the backend configuration that the model runs with.
- `usage`: the number of tokens used to generate the replies, counting prompt, completion, and total

Extract just the reply with:


```python
response.choices[0].message.content

```




    'Orange who?'



Even non-conversation-based tasks can fit into the chat format, by placing the instruction in the first user message.

For example, to ask the model to explain asynchronous programming in the style of the pirate Blackbeard, we can structure conversation as follows:


```python
# example with a system message
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain asynchronous programming in the style of the pirate Blackbeard."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    Arr, me matey! Let me tell ye a tale of asynchronous programming, in the style of the fearsome pirate Blackbeard!
    
    Picture this, me hearties. In the vast ocean of programming, there be times when ye need to perform multiple tasks at once. But fear not, for asynchronous programming be here to save the day!
    
    Ye see, in traditional programming, ye be waitin' for one task to be done before movin' on to the next. But with asynchronous programming, ye can be takin' care of multiple tasks at the same time, just like a pirate multitaskin' on the high seas!
    
    Instead of waitin' for a task to be completed, ye can be sendin' it off on its own journey, while ye move on to the next task. It be like havin' a crew of trusty sailors, each takin' care of their own duties, without waitin' for the others.
    
    Now, ye may be wonderin', how does this sorcery work? Well, me matey, it be all about callbacks and promises. When ye be sendin' off a task, ye be attachin' a callback function to it. This be like leavin' a message in a bottle, tellin' the task what to do when it be finished.
    
    While the task be sailin' on its own, ye can be movin' on to the next task, without wastin' any precious time. And when the first task be done, it be sendin' a signal back to ye, lettin' ye know it be finished. Then ye can be takin' care of the callback function, like openin' the bottle and readin' the message inside.
    
    But wait, there be more! With promises, ye can be makin' even fancier arrangements. Instead of callbacks, ye be makin' a promise that the task will be completed. It be like a contract between ye and the task, swearin' that it will be done.
    
    Ye can be attachin' multiple promises to a task, promisin' different outcomes. And when the task be finished, it be fulfillin' the promises, lettin' ye know it be done. Then ye can be handlin' the fulfillments, like collectin' the rewards of yer pirate adventures!
    
    So, me hearties, that be the tale of asynchronous programming, told in the style of the fearsome pirate Blackbeard! With callbacks and promises, ye can be takin' care of multiple tasks at once, just like a pirate conquerin' the seven seas!


key: _node_content
string_value:
{"id_": "f61fa2ba-d8a0-4210-b298-7f4e19dbf4c7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56b9e192-f132-44c6-b884-6fbcf4552789", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ff6633b713086fc185e4cf37795228bc326c9a8401e9eb49ef7762f4e48f2d6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05e80a33-7263-44d8-9d06-be5a4cedeff5", "node_type": "1", "metadata": {}, "hash": "45bef14035f7ebe9ef30490d89d70fc3ef99c6cc66668d446872aede9da11558", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8548, "end_char_idx": 11484, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# example without a system message
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "user", "content": "Explain asynchronous programming in the style of the pirate Blackbeard."},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    Arr, me hearties! Gather 'round and listen up, for I be tellin' ye about the mysterious art of asynchronous programming, in the style of the fearsome pirate Blackbeard!
    
    Now, ye see, in the world of programming, there be times when we need to perform tasks that take a mighty long time to complete. These tasks might involve fetchin' data from the depths of the internet, or performin' complex calculations that would make even Davy Jones scratch his head.
    
    In the olden days, we pirates used to wait patiently for each task to finish afore movin' on to the next one. But that be a waste of precious time, me hearties! We be pirates, always lookin' for ways to be more efficient and plunder more booty!
    
    That be where asynchronous programming comes in, me mateys. It be a way to tackle multiple tasks at once, without waitin' for each one to finish afore movin' on. It be like havin' a crew of scallywags workin' on different tasks simultaneously, while ye be overseein' the whole operation.
    
    Ye see, in asynchronous programming, we be breakin' down our tasks into smaller chunks called "coroutines." Each coroutine be like a separate pirate, workin' on its own task. When a coroutine be startin' its work, it don't wait for the task to finish afore movin' on to the next one. Instead, it be movin' on to the next task, lettin' the first one continue in the background.
    
    Now, ye might be wonderin', "But Blackbeard, how be we know when a task be finished if we don't wait for it?" Ah, me hearties, that be where the magic of callbacks and promises come in!
    
    When a coroutine be startin' its work, it be attachin' a callback or a promise to it. This be like leavin' a message in a bottle, tellin' the coroutine what to do when it be finished. So, while the coroutine be workin' away, the rest of the crew be movin' on to other tasks, plunderin' more booty along the way.
    
    When a coroutine be finished with its task, it be sendin' a signal to the callback or fulfillin' the promise, lettin' the rest of the crew know that it be done. Then, the crew can gather 'round and handle the results of the completed task, celebratin' their victory and countin' their plunder.
    
    So, me hearties, asynchronous programming be like havin' a crew of pirates workin' on different tasks at once, without waitin' for each one to finish afore movin' on. It be a way to be more efficient, plunder more booty, and conquer the vast seas of programming!
    
    Now, set sail, me mateys, and embrace the power of asynchronous programming like true pirates of the digital realm! Arr!


key: _node_content
string_value:
{"id_": "05e80a33-7263-44d8-9d06-be5a4cedeff5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "97c1d8b6-cf24-4e05-8d76-e5db80ea6f05", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7c48481b2944edc8213def44183cf00818458397e592cb1568a4cec646d5cf79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f61fa2ba-d8a0-4210-b298-7f4e19dbf4c7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb", "title": "How to format inputs to ChatGPT models", "tokens": 5687, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "144774a3450dd3bafef8e372cf727f1f16cd6920fc31d4a684927182d3e06874", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b59f247d-59c6-46af-9d0a-d0c248f0a3d6", "node_type": "1", "metadata": {}, "hash": "60833d3ef16ded3c94b9de0fa1c2cc6934fb80a09b955d439522768b96d9b24e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11487, "end_char_idx": 14789, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 3. Tips for instructing gpt-3.5-turbo-0301

Best practices for instructing models may change from model version to model version. The advice that follows applies to `gpt-3.5-turbo-0301` and may not apply to future models.

### System messages

The system message can be used to prime the assistant with different personalities or behaviors.

Be aware that `gpt-3.5-turbo-0301` does not generally pay as much attention to the system message as `gpt-4-0314` or `gpt-3.5-turbo-0613`. Therefore, for `gpt-3.5-turbo-0301`, we recommend placing important instructions in the user message instead. Some developers have found success in continually moving the system message near the end of the conversation to keep the model's attention from drifting away as conversations get longer.


```python
# An example of a system message that primes the assistant to explain concepts in great depth
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a friendly and helpful teaching assistant. You explain concepts in great depth using simple terms, and you give examples to help people learn. At the end of each explanation, you ask a question to check for understanding"},
        {"role": "user", "content": "Can you explain how fractions work?"},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    Of course! Fractions are a way to represent parts of a whole. They are made up of two numbers: a numerator and a denominator. The numerator tells you how many parts you have, and the denominator tells you how many equal parts make up the whole.
    
    Let's take an example to understand this better. Imagine you have a pizza that is divided into 8 equal slices. If you eat 3 slices, you can represent that as the fraction 3/8. Here, the numerator is 3 because you ate 3 slices, and the denominator is 8 because the whole pizza is divided into 8 slices.
    
    Fractions can also be used to represent numbers less than 1. For example, if you eat half of a pizza, you can write it as 1/2. Here, the numerator is 1 because you ate one slice, and the denominator is 2 because the whole pizza is divided into 2 equal parts.
    
    Now, let's talk about equivalent fractions. Equivalent fractions are different fractions that represent the same amount. For example, 1/2 and 2/4 are equivalent fractions because they both represent half of something. To find equivalent fractions, you can multiply or divide both the numerator and denominator by the same number.
    
    Here's a question to check your understanding: If you have a cake divided into 12 equal slices and you eat 4 slices, what fraction of the cake did you eat?



```python
# An example of a system message that primes the assistant to give brief, to-the-point answers
response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a laconic assistant. You reply with brief, to-the-point answers with no elaboration."},
        {"role": "user", "content": "Can you explain how fractions work?"},
    ],
    temperature=0,
)

print(response.choices[0].message.content)

```

    Fractions represent parts of a whole. They have a numerator (top number) and a denominator (bottom number).


