# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 78a6cd18-b603-46b6-8f75-0f42f4b746b2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"78a6cd18-b603-46b6-8f75-0f42f4b746b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"19c5aa20-02f8-484e-bac3-e55a706996cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"090c8a15bb6567b0e145953d5f6c86a3e6efae9385a51a1cdcf228d2482edacd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3065, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "ref_doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "chroma:document": "# Embedding texts that are longer than the model's maximum context length\n\nOpenAI's embedding models cannot embed text that exceeds a maximum length. The maximum length varies by model, and is measured by _tokens_, not string length. If you are unfamiliar with tokenization, check out [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).\n\nThis notebook shows how to handle texts that are longer than a model's maximum context length. We'll demonstrate using embeddings from `text-embedding-3-small`, but the same ideas can be applied to other models and tasks. To learn more about embeddings, check out the [OpenAI Embeddings Guide](https://beta.openai.com/docs/guides/embeddings).\n\n\n## 1. Model context length\n\nFirst, we select the model and define a function to get embeddings from the API.\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport openai\nfrom tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_not_exception_type\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\nEMBEDDING_MODEL = 'text-embedding-3-small'\nEMBEDDING_CTX_LENGTH = 8191\nEMBEDDING_ENCODING = 'cl100k_base'\n\n# let's make sure to not retry on an invalid request, because that is what we want to demonstrate\n@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6), retry=retry_if_not_exception_type(openai.BadRequestError))\ndef get_embedding(text_or_tokens, model=EMBEDDING_MODEL):\n    return client.embeddings.create(input=text_or_tokens, model=model).data[0].embedding\n```\n\nThe `text-embedding-3-small` model has a context length of 8191 tokens with the `cl100k_base` encoding, and we can see that going over that limit causes an error.\n\n\n```python\nlong_text = 'AGI ' * 5000\ntry:\n    get_embedding(long_text)\nexcept openai.BadRequestError as e:\n    print(e)\n```\n\n    Error code: 400 - {'error': {'message': \"This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.\", 'type': 'invalid_request_error', 'param': None, 'code': None}}\n\n\nClearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.\n\n## 1. Truncating the input text\n\nThe simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"19c5aa20-02f8-484e-bac3-e55a706996cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78a6cd18-b603-46b6-8f75-0f42f4b746b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e31f8190e60166d6e11390c70a316532b0f0b181a6df4930b1db8c79c5795c00\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1f21c57a-b558-4416-8989-aa9a6ea88946\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"de80614b0c643df13c462169da1ca86aeab029ddab624490443291a20c5f9a9b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1749, \"end_char_idx\": 5132, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "ref_doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "chroma:document": "```python\nlong_text = 'AGI ' * 5000\ntry:\n    get_embedding(long_text)\nexcept openai.BadRequestError as e:\n    print(e)\n```\n\n    Error code: 400 - {'error': {'message': \"This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.\", 'type': 'invalid_request_error', 'param': None, 'code': None}}\n\n\nClearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.\n\n## 1. Truncating the input text\n\nThe simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.\n\n\n```python\nimport tiktoken\n\ndef truncate_text_tokens(text, encoding_name=EMBEDDING_ENCODING, max_tokens=EMBEDDING_CTX_LENGTH):\n    \"\"\"Truncate a string to have `max_tokens` according to the given encoding.\"\"\"\n    encoding = tiktoken.get_encoding(encoding_name)\n    return encoding.encode(text)[:max_tokens]\n```\n\nOur example from before now works without error.\n\n\n```python\ntruncated = truncate_text_tokens(long_text)\nlen(get_embedding(truncated))\n```\n\n\n\n\n    1536\n\n\n\n## 2. Chunking the input text\n\nThough truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).\n\nWe will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.\n\n\n```python\nfrom itertools import islice\n\ndef batched(iterable, n):\n    \"\"\"Batch data into tuples of length n. The last batch may be shorter.\"\"\"\n    # batched('ABCDEFG', 3) --> ABC DEF G\n    if n < 1:\n        raise ValueError('n must be at least one')\n    it = iter(iterable)\n    while (batch := tuple(islice(it, n))):\n        yield batch\n```\n\nNow we define a function that encodes a string into tokens and then breaks it up into chunks.\n\n\n```python\ndef chunked_tokens(text, encoding_name, chunk_length):\n    encoding = tiktoken.get_encoding(encoding_name)\n    tokens = encoding.encode(text)\n    chunks_iterator = batched(tokens, chunk_length)\n    yield from chunks_iterator\n```\n\nFinally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.\n\n\n```python\nimport numpy as np"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1f21c57a-b558-4416-8989-aa9a6ea88946
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1f21c57a-b558-4416-8989-aa9a6ea88946\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"19c5aa20-02f8-484e-bac3-e55a706996cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb\", \"title\": \"Embedding texts that are longer than the model's maximum context length\", \"tokens\": 1474, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d20efe88cb3ac782fff136879b5782ef5756adc7524499ce751456eb4ec4f0a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3430, \"end_char_idx\": 6595, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "ref_doc_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "chroma:document": "```python\ntruncated = truncate_text_tokens(long_text)\nlen(get_embedding(truncated))\n```\n\n\n\n\n    1536\n\n\n\n## 2. Chunking the input text\n\nThough truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).\n\nWe will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.\n\n\n```python\nfrom itertools import islice\n\ndef batched(iterable, n):\n    \"\"\"Batch data into tuples of length n. The last batch may be shorter.\"\"\"\n    # batched('ABCDEFG', 3) --> ABC DEF G\n    if n < 1:\n        raise ValueError('n must be at least one')\n    it = iter(iterable)\n    while (batch := tuple(islice(it, n))):\n        yield batch\n```\n\nNow we define a function that encodes a string into tokens and then breaks it up into chunks.\n\n\n```python\ndef chunked_tokens(text, encoding_name, chunk_length):\n    encoding = tiktoken.get_encoding(encoding_name)\n    tokens = encoding.encode(text)\n    chunks_iterator = batched(tokens, chunk_length)\n    yield from chunks_iterator\n```\n\nFinally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.\n\n\n```python\nimport numpy as np\n\n\ndef len_safe_get_embedding(text, model=EMBEDDING_MODEL, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING, average=True):\n    chunk_embeddings = []\n    chunk_lens = []\n    for chunk in chunked_tokens(text, encoding_name=encoding_name, chunk_length=max_tokens):\n        chunk_embeddings.append(get_embedding(chunk, model=model))\n        chunk_lens.append(len(chunk))\n\n    if average:\n        chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)\n        chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)  # normalizes length to 1\n        chunk_embeddings = chunk_embeddings.tolist()\n    return chunk_embeddings\n```\n\nOnce again, we can now handle long input texts.\n\n\n```python\naverage_embedding_vector = len_safe_get_embedding(long_text, average=True)\nchunks_embedding_vectors = len_safe_get_embedding(long_text, average=False)\n\nprint(f\"Setting average=True gives us a single {len(average_embedding_vector)}-dimensional embedding vector for our long text.\")\nprint(f\"Setting average=False gives us {len(chunks_embedding_vectors)} embedding vectors, one for each of the chunks.\")\n\n```\n\n    Setting average=True gives us a single 1536-dimensional embedding vector for our long text.\n    Setting average=False gives us 2 embedding vectors, one for each of the chunks.\n\n\nIn some cases, it may make sense to split chunks on paragraph boundaries or sentence boundaries to help preserve the meaning of the text."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e331bb7a-6d7f-4349-811d-d9cbf3449891
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e331bb7a-6d7f-4349-811d-d9cbf3449891\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04415a1c-4881-4738-aa11-a94ead8b6217\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb8cfe9991ab81df4370da4d754f071a628627c2cb5102099bbc10871f3f8a63\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2533, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "# Summarizing Long Documents\n\nThe objective of this notebook is to demonstrate how to summarize large documents with a controllable level of detail.\n \nIf you give a GPT model the task of summarizing a long document (e.g. 10k or more tokens), you'll tend to get back a relatively short summary that isn't proportional to the length of the document. For instance, a summary of a 20k token document will not be twice as long as a summary of a 10k token document. One way we can fix this is to split our document up into pieces, and produce a summary piecewise. After many queries to a GPT model, the full summary can be reconstructed. By controlling the number of text chunks and their sizes, we can ultimately control the level of detail in the output.\n\n\n```python\nimport os\nfrom typing import List, Tuple, Optional\nfrom openai import OpenAI\nimport tiktoken\nfrom tqdm import tqdm\n```\n\n\n```python\n# open dataset containing part of the text of the Wikipedia page for the United States\nwith open(\"data/artificial_intelligence_wikipedia.txt\", \"r\") as file:\n    artificial_intelligence_wikipedia_text = file.read()\n```\n\n\n```python\n# load encoding and check the length of dataset\nencoding = tiktoken.encoding_for_model('gpt-4-turbo')\nlen(encoding.encode(artificial_intelligence_wikipedia_text))\n```\n\n\n\n\n    14630\n\n\n\nWe'll define a simple utility to wrap calls to the OpenAI API.\n\n\n```python\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\ndef get_chat_completion(messages, model='gpt-4-turbo'):\n    response = client.chat.completions.create(\n        model=model,\n        messages=messages,\n        temperature=0,\n    )\n    return response.choices[0].message.content\n```\n\nNext we'll define some utilities to chunk a large document into smaller pieces.\n\n\n```python\ndef tokenize(text: str) -> List[str]:\n    encoding = tiktoken.encoding_for_model('gpt-4-turbo')\n    return encoding.encode(text)\n\n\n# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.\ndef chunk_on_delimiter(input_string: str,\n                       max_tokens: int, delimiter: str) -> List[str]:\n    chunks = input_string.split(delimiter)\n    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(\n        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True\n    )\n    if dropped_chunk_count > 0:\n        print(f\"warning: {dropped_chunk_count} chunks were dropped due to overflow\")\n    combined_chunks = [f\"{chunk}{delimiter}\" for chunk in combined_chunks]\n    return combined_chunks"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 04415a1c-4881-4738-aa11-a94ead8b6217
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"04415a1c-4881-4738-aa11-a94ead8b6217\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e331bb7a-6d7f-4349-811d-d9cbf3449891\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a080324b075109cf3df5514cfcff4084bf36ea45f42a7703ae513a9ab57bbdaa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7b915b4a-3687-4141-9aef-08a052a8857c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"530d5258402ea19b0dd2cb1e2e3ed5ec1fffbeddd6ba8a9695fef733fdbb0487\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 884, \"end_char_idx\": 5207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "```python\n# open dataset containing part of the text of the Wikipedia page for the United States\nwith open(\"data/artificial_intelligence_wikipedia.txt\", \"r\") as file:\n    artificial_intelligence_wikipedia_text = file.read()\n```\n\n\n```python\n# load encoding and check the length of dataset\nencoding = tiktoken.encoding_for_model('gpt-4-turbo')\nlen(encoding.encode(artificial_intelligence_wikipedia_text))\n```\n\n\n\n\n    14630\n\n\n\nWe'll define a simple utility to wrap calls to the OpenAI API.\n\n\n```python\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\ndef get_chat_completion(messages, model='gpt-4-turbo'):\n    response = client.chat.completions.create(\n        model=model,\n        messages=messages,\n        temperature=0,\n    )\n    return response.choices[0].message.content\n```\n\nNext we'll define some utilities to chunk a large document into smaller pieces.\n\n\n```python\ndef tokenize(text: str) -> List[str]:\n    encoding = tiktoken.encoding_for_model('gpt-4-turbo')\n    return encoding.encode(text)\n\n\n# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.\ndef chunk_on_delimiter(input_string: str,\n                       max_tokens: int, delimiter: str) -> List[str]:\n    chunks = input_string.split(delimiter)\n    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(\n        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True\n    )\n    if dropped_chunk_count > 0:\n        print(f\"warning: {dropped_chunk_count} chunks were dropped due to overflow\")\n    combined_chunks = [f\"{chunk}{delimiter}\" for chunk in combined_chunks]\n    return combined_chunks\n\n\n# This function combines text chunks into larger blocks without exceeding a specified token count. It returns the combined text blocks, their original indices, and the count of chunks dropped due to overflow.\ndef combine_chunks_with_no_minimum(\n        chunks: List[str],\n        max_tokens: int,\n        chunk_delimiter=\"\\n\\n\",\n        header: Optional[str] = None,\n        add_ellipsis_for_overflow=False,\n) -> Tuple[List[str], List[int]]:\n    dropped_chunk_count = 0\n    output = []  # list to hold the final combined chunks\n    output_indices = []  # list to hold the indices of the final combined chunks\n    candidate = (\n        [] if header is None else [header]\n    )  # list to hold the current combined chunk candidate\n    candidate_indices = []\n    for chunk_i, chunk in enumerate(chunks):\n        chunk_with_header = [chunk] if header is None else [header, chunk]\n        if len(tokenize(chunk_delimiter.join(chunk_with_header))) > max_tokens:\n            print(f\"warning: chunk overflow\")\n            if (\n                    add_ellipsis_for_overflow\n                    and len(tokenize(chunk_delimiter.join(candidate + [\"...\"]))) <= max_tokens\n            ):\n                candidate.append(\"...\")\n                dropped_chunk_count += 1\n            continue  # this case would break downstream assumptions\n        # estimate token count with the current chunk added\n        extended_candidate_token_count = len(tokenize(chunk_delimiter.join(candidate + [chunk])))\n        # If the token count exceeds max_tokens, add the current candidate to output and start a new candidate\n        if extended_candidate_token_count > max_tokens:\n            output.append(chunk_delimiter.join(candidate))\n            output_indices.append(candidate_indices)\n            candidate = chunk_with_header  # re-initialize candidate\n            candidate_indices = [chunk_i]\n        # otherwise keep extending the candidate\n        else:\n            candidate.append(chunk)\n            candidate_indices.append(chunk_i)\n    # add the remaining candidate to output if it's not empty\n    if (header is not None and len(candidate) > 1) or (header is None and len(candidate) > 0):\n        output.append(chunk_delimiter.join(candidate))\n        output_indices.append(candidate_indices)\n    return output, output_indices, dropped_chunk_count\n```\n\nNow we can define a utility to summarize text with a controllable level of detail (note the `detail` parameter).\n\nThe function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on a controllable `detail` parameter. It then splits the text into chunks and summarizes each chunk."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7b915b4a-3687-4141-9aef-08a052a8857c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7b915b4a-3687-4141-9aef-08a052a8857c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"04415a1c-4881-4738-aa11-a94ead8b6217\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2a7821aa039b27f57dce0dec91d94f1927d384d38fa4f067f8205c40eec794fc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8ef89ce9-578a-4d2f-8195-4690e41644dc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b0b2d19746c9828ea854df7a96485e12429bff1f0c98a2d7f011f348428a8e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5210, \"end_char_idx\": 8032, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "```python\ndef summarize(text: str,\n              detail: float = 0,\n              model: str = 'gpt-4-turbo',\n              additional_instructions: Optional[str] = None,\n              minimum_chunk_size: Optional[int] = 500,\n              chunk_delimiter: str = \".\",\n              summarize_recursively=False,\n              verbose=False):\n    \"\"\"\n    Summarizes a given text by splitting it into chunks, each of which is summarized individually. \n    The level of detail in the summary can be adjusted, and the process can optionally be made recursive.\n\n    Parameters:\n    - text (str): The text to be summarized.\n    - detail (float, optional): A value between 0 and 1 indicating the desired level of detail in the summary.\n      0 leads to a higher level summary, and 1 results in a more detailed summary. Defaults to 0.\n    - model (str, optional): The model to use for generating summaries. Defaults to 'gpt-3.5-turbo'.\n    - additional_instructions (Optional[str], optional): Additional instructions to provide to the model for customizing summaries.\n    - minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.\n    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to \".\".\n    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.\n    - verbose (bool, optional): If True, prints detailed information about the chunking process.\n\n    Returns:\n    - str: The final compiled summary of the text.\n\n    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. \n    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, \n    adding more context to the summarization process. The function returns a compiled summary of all chunks.\n    \"\"\"\n\n    # check detail is set correctly\n    assert 0 <= detail <= 1\n\n    # interpolate the number of chunks based to get specified level of detail\n    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))\n    min_chunks = 1\n    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))\n\n    # adjust chunk_size based on interpolated number of chunks\n    document_length = len(tokenize(text))\n    chunk_size = max(minimum_chunk_size, document_length // num_chunks)\n    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)\n    if verbose:\n        print(f\"Splitting the text into {len(text_chunks)} chunks to be summarized.\")\n        print(f\"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}\")\n\n    # set system message\n    system_message_content = \"Rewrite this text in summarized form.\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8ef89ce9-578a-4d2f-8195-4690e41644dc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8ef89ce9-578a-4d2f-8195-4690e41644dc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7b915b4a-3687-4141-9aef-08a052a8857c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4b5cbd081b9e40a19f9865d380f03cff6b74b2353ab7b00adc8f3e53a835f85b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a47eeb53-6747-4a0e-8a4b-a2bdca928351\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"378f3360a653a69e4e04a78c726e171eab26bdb4a6557620254c821d5642ced0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6273, \"end_char_idx\": 9895, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "- minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.\n    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to \".\".\n    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.\n    - verbose (bool, optional): If True, prints detailed information about the chunking process.\n\n    Returns:\n    - str: The final compiled summary of the text.\n\n    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. \n    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, \n    adding more context to the summarization process. The function returns a compiled summary of all chunks.\n    \"\"\"\n\n    # check detail is set correctly\n    assert 0 <= detail <= 1\n\n    # interpolate the number of chunks based to get specified level of detail\n    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))\n    min_chunks = 1\n    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))\n\n    # adjust chunk_size based on interpolated number of chunks\n    document_length = len(tokenize(text))\n    chunk_size = max(minimum_chunk_size, document_length // num_chunks)\n    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)\n    if verbose:\n        print(f\"Splitting the text into {len(text_chunks)} chunks to be summarized.\")\n        print(f\"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}\")\n\n    # set system message\n    system_message_content = \"Rewrite this text in summarized form.\"\n    if additional_instructions is not None:\n        system_message_content += f\"\\n\\n{additional_instructions}\"\n\n    accumulated_summaries = []\n    for chunk in tqdm(text_chunks):\n        if summarize_recursively and accumulated_summaries:\n            # Creating a structured prompt for recursive summarization\n            accumulated_summaries_string = '\\n\\n'.join(accumulated_summaries)\n            user_message_content = f\"Previous summaries:\\n\\n{accumulated_summaries_string}\\n\\nText to summarize next:\\n\\n{chunk}\"\n        else:\n            # Directly passing the chunk for summarization without recursive context\n            user_message_content = chunk\n\n        # Constructing messages based on whether recursive summarization is applied\n        messages = [\n            {\"role\": \"system\", \"content\": system_message_content},\n            {\"role\": \"user\", \"content\": user_message_content}\n        ]\n\n        # Assuming this function gets the completion and works as expected\n        response = get_chat_completion(messages, model=model)\n        accumulated_summaries.append(response)\n\n    # Compile final summary from partial summaries\n    final_summary = '\\n\\n'.join(accumulated_summaries)\n\n    return final_summary\n```\n\nNow we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.\n\n\n```python\nsummary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)\n```\n\n    Splitting the text into 1 chunks to be summarized.\n    Chunk lengths are [14631]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a47eeb53-6747-4a0e-8a4b-a2bdca928351
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a47eeb53-6747-4a0e-8a4b-a2bdca928351\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8ef89ce9-578a-4d2f-8195-4690e41644dc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8ba713e0f62ef805d307226bc3460ca8d011e56b491e75f003a6edda66874717\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2fab2cde-2515-46ac-ad29-c5b408c52b6a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"908d2655849b7cddf0fd54145785f377d022fe614ffb8000b0323ab4a3dd4966\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8037, \"end_char_idx\": 10590, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "if additional_instructions is not None:\n        system_message_content += f\"\\n\\n{additional_instructions}\"\n\n    accumulated_summaries = []\n    for chunk in tqdm(text_chunks):\n        if summarize_recursively and accumulated_summaries:\n            # Creating a structured prompt for recursive summarization\n            accumulated_summaries_string = '\\n\\n'.join(accumulated_summaries)\n            user_message_content = f\"Previous summaries:\\n\\n{accumulated_summaries_string}\\n\\nText to summarize next:\\n\\n{chunk}\"\n        else:\n            # Directly passing the chunk for summarization without recursive context\n            user_message_content = chunk\n\n        # Constructing messages based on whether recursive summarization is applied\n        messages = [\n            {\"role\": \"system\", \"content\": system_message_content},\n            {\"role\": \"user\", \"content\": user_message_content}\n        ]\n\n        # Assuming this function gets the completion and works as expected\n        response = get_chat_completion(messages, model=model)\n        accumulated_summaries.append(response)\n\n    # Compile final summary from partial summaries\n    final_summary = '\\n\\n'.join(accumulated_summaries)\n\n    return final_summary\n```\n\nNow we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.\n\n\n```python\nsummary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)\n```\n\n    Splitting the text into 1 chunks to be summarized.\n    Chunk lengths are [14631]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1/1 [00:09<00:00,  9.68s/it]\n\n\n\n```python\nsummary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)\n```\n\n    Splitting the text into 9 chunks to be summarized.\n    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 9/9 [01:33<00:00, 10.39s/it]\n\n\n\n```python\nsummary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)\n```\n\n    Splitting the text into 17 chunks to be summarized.\n    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 17/17 [02:26<00:00,  8.64s/it]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2fab2cde-2515-46ac-ad29-c5b408c52b6a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2fab2cde-2515-46ac-ad29-c5b408c52b6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a47eeb53-6747-4a0e-8a4b-a2bdca928351\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"420d01d8a01fa0c3a12939668bc012fcd9cca737a5e9165433a627005ed9cdfb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0665041-96dc-4ddc-be7b-cc42a8a67a17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4731f1bef86681130c29ed3d6986c127b10be23a49b5f4056c5089294c13e07d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9339, \"end_char_idx\": 11499, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.\n\n\n```python\nsummary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)\n```\n\n    Splitting the text into 1 chunks to be summarized.\n    Chunk lengths are [14631]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1/1 [00:09<00:00,  9.68s/it]\n\n\n\n```python\nsummary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)\n```\n\n    Splitting the text into 9 chunks to be summarized.\n    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 9/9 [01:33<00:00, 10.39s/it]\n\n\n\n```python\nsummary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)\n```\n\n    Splitting the text into 17 chunks to be summarized.\n    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 17/17 [02:26<00:00,  8.64s/it]\n\n\n\n```python\nsummary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)\n```\n\n    Splitting the text into 31 chunks to be summarized.\n    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 31/31 [04:08<00:00,  8.02s/it]\n\n\nThe original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!\n\n\n```python\n# lengths of summaries\n[len(tokenize(x)) for x in\n [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]\n```\n\n\n\n\n    [235, 2529, 4336, 6742]\n\n\n\nLet's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f0665041-96dc-4ddc-be7b-cc42a8a67a17
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f0665041-96dc-4ddc-be7b-cc42a8a67a17\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2fab2cde-2515-46ac-ad29-c5b408c52b6a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4dccf5e0b00dba222b9836d80f72c90dcb2e826750e3c5d8e5dda2d8e4cf05d8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"66ae92f4-3af8-4095-83dc-70838673f972\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"998d98a9b01514c8e84d02fc8620f30e0342a3a681bf5bf8279c2a27a8396869\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10543, \"end_char_idx\": 13699, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 17/17 [02:26<00:00,  8.64s/it]\n\n\n\n```python\nsummary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)\n```\n\n    Splitting the text into 31 chunks to be summarized.\n    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 31/31 [04:08<00:00,  8.02s/it]\n\n\nThe original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!\n\n\n```python\n# lengths of summaries\n[len(tokenize(x)) for x in\n [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]\n```\n\n\n\n\n    [235, 2529, 4336, 6742]\n\n\n\nLet's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1.\n\n\n```python\nprint(summary_with_detail_0)\n```\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as \"AI winters.\" Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.\n    \n    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.\n    \n    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.\n\n\n\n```python\nprint(summary_with_detail_1)\n```\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.\n    \n    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 66ae92f4-3af8-4095-83dc-70838673f972
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"66ae92f4-3af8-4095-83dc-70838673f972\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0665041-96dc-4ddc-be7b-cc42a8a67a17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7f3483b43de6a9b3ecdbb5f689742223ce811c6f7fdeeb2c02f9377125c3505a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"316fcdbb-9774-469d-bf11-6d54ea1092c9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b10de7fa67b0fd56ee28c6a0d28ad4563374c164af648f70b609513c73e0db4c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11502, \"end_char_idx\": 15642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "```python\nprint(summary_with_detail_0)\n```\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as \"AI winters.\" Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.\n    \n    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.\n    \n    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.\n\n\n\n```python\nprint(summary_with_detail_1)\n```\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.\n    \n    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.\n    \n    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.\n    \n    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.\n    \n    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 316fcdbb-9774-469d-bf11-6d54ea1092c9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"316fcdbb-9774-469d-bf11-6d54ea1092c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"66ae92f4-3af8-4095-83dc-70838673f972\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e21cdd7a8a285e32be00fc02bcc5997ea6e66da59e1df2e41ac8b5c5f5222213\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a97e45f8-3426-42f1-9e37-64f87e3057f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f4d6b6a3dd94873d90ba96789cda548b19b210387d9b841aa31f54d36c63bff2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13535, \"end_char_idx\": 17775, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.\n    \n    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.\n    \n    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.\n    \n    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.\n    \n    In the realm of AI, an \"agent\" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.\n    \n    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.\n    \n    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.\n    \n    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a97e45f8-3426-42f1-9e37-64f87e3057f0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a97e45f8-3426-42f1-9e37-64f87e3057f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"316fcdbb-9774-469d-bf11-6d54ea1092c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"471668d818f4db193d71fff39182fa1766c16c6e876aca1debae1e3e9c286698\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0450bc52-2f37-4417-b5cf-a88f150ca175\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0242685a2ea8c8ab7697bf165aacee0e9bb71076b7a908e61d6b7d32d9dfb6e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15537, \"end_char_idx\": 19778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.\n    \n    In the realm of AI, an \"agent\" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.\n    \n    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.\n    \n    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.\n    \n    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.\n    \n    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.\n    \n    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.\n    \n    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0450bc52-2f37-4417-b5cf-a88f150ca175
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0450bc52-2f37-4417-b5cf-a88f150ca175\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a97e45f8-3426-42f1-9e37-64f87e3057f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"54aa6fbec83ee4d15c5ff94aa9c207f89c98ada7bf82e60ce14d2bf0aafbf675\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b1e5a906-c515-415e-ac1d-b080b4b82648\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab23f571cb31e4b952ac34a60d584b3fb9e6d62c72a118544289419bbcad9fc0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17611, \"end_char_idx\": 21774, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.\n    \n    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.\n    \n    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.\n    \n    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.\n    \n    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.\n    \n    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.\n    \n    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.\n    \n    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.\n    \n    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b1e5a906-c515-415e-ac1d-b080b4b82648
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b1e5a906-c515-415e-ac1d-b080b4b82648\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0450bc52-2f37-4417-b5cf-a88f150ca175\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce582ff42160f14dff35edea0f13b212e0e781128b19115fa7256396257e2d78\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"176b127e-4efb-4ffb-adaa-80eb9cbf768c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d1a7f143795073479fbed1c2faaac497692c027ab1a6b1fe5f5baa3b333bb4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19637, \"end_char_idx\": 23695, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.\n    \n    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.\n    \n    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.\n    \n    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.\n    \n    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.\n    \n    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.\n    \n    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.\n    \n    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.\n    \n    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 176b127e-4efb-4ffb-adaa-80eb9cbf768c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"176b127e-4efb-4ffb-adaa-80eb9cbf768c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1e5a906-c515-415e-ac1d-b080b4b82648\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f593a5961d0e0d8c83bdd68df9f59385e7d9b64f8e0c4635d8eb1290cd2971d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b03416f-f813-44e0-ad39-7205f88cf503\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3d1ff00cf2584d17160f4c3eed3caf124cb296b04298ed6f9fdca519076674a6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21613, \"end_char_idx\": 25614, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.\n    \n    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.\n    \n    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.\n    \n    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text. These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.\n    \n    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as \"hallucinations.\" These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.\n    \n    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.\n    \n    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.\n    \n    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.\n    \n    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0b03416f-f813-44e0-ad39-7205f88cf503
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0b03416f-f813-44e0-ad39-7205f88cf503\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"176b127e-4efb-4ffb-adaa-80eb9cbf768c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"b7c2bba2975b17f2893be30e5794f0e50c1571f0b8b07d7405251c0cbc4319fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f904e105-2a91-4560-9b5f-c85aaf778693\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c7179dbc8978aea81bcc3dc3325bb69e5e67008658c294fec835721483b912df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23696, \"end_char_idx\": 27670, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.\n    \n    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as \"hallucinations.\" These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.\n    \n    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.\n    \n    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.\n    \n    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.\n    \n    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria. In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.\n    \n    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.\n    \n    Ethics and Risks of AI\n    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.\n    \n    Privacy and Copyright Issues\n    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f904e105-2a91-4560-9b5f-c85aaf778693
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f904e105-2a91-4560-9b5f-c85aaf778693\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b03416f-f813-44e0-ad39-7205f88cf503\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"202b87df5a8b20a41ca35f1ce8fd6122cb7c17e50e0b89331b77311c4fbc373e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f94fbbec-0bad-4938-bb51-e9020cd7c177\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e8921eb4d40318745357355d26cd21f6cd85957030083d19ca97b6c7e1b06161\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25615, \"end_char_idx\": 29584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.\n    \n    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.\n    \n    Ethics and Risks of AI\n    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.\n    \n    Privacy and Copyright Issues\n    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.\n    \n    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming \"fair use.\" The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.\n    \n    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.\n    \n    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.\n    \n    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.\n    \n    In 2015, Google Photos mislabeled Jacky Alcine and his friend as \"gorillas\" due to a lack of diverse images in its training dataset, an issue known as \"sample size disparity.\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f94fbbec-0bad-4938-bb51-e9020cd7c177
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f94fbbec-0bad-4938-bb51-e9020cd7c177\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f904e105-2a91-4560-9b5f-c85aaf778693\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"96e610ef3669ab9d15d4e83b9a54db6f8b489b4d17d34a76f7e9c96f15524929\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e3ad7667-7713-41da-939e-ebe5df86f4b7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"86d9a8e797aebe8f85e4955e6f8def04f794794cbefaa9513db9f90d9c04bbae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27512, \"end_char_idx\": 31549, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.\n    \n    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming \"fair use.\" The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.\n    \n    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.\n    \n    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.\n    \n    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.\n    \n    In 2015, Google Photos mislabeled Jacky Alcine and his friend as \"gorillas\" due to a lack of diverse images in its training dataset, an issue known as \"sample size disparity.\" Google's temporary solution was to stop labeling any images as \"gorilla,\" a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.\n    \n    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.\n    \n    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e3ad7667-7713-41da-939e-ebe5df86f4b7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e3ad7667-7713-41da-939e-ebe5df86f4b7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f94fbbec-0bad-4938-bb51-e9020cd7c177\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8d0c4f7017ef3978e1db9f60dc1d73ff27efe88b36545ec8fd7000b4f243697\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"009e71b1-981f-4fa5-b760-c4aa5a131c90\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d59b58e65dcc4f97123cd2a6f957e868c0d00245e5f78ccca6909a3f61acebf1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29585, \"end_char_idx\": 33690, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "Google's temporary solution was to stop labeling any images as \"gorilla,\" a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.\n    \n    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.\n    \n    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.\n    \n    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.\n    \n    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.\n    \n    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 009e71b1-981f-4fa5-b760-c4aa5a131c90
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"009e71b1-981f-4fa5-b760-c4aa5a131c90\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e3ad7667-7713-41da-939e-ebe5df86f4b7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ed6c1ad9967c001d81bc9c3a8b477c98862bbc6c7866bb64803486dccac57cc0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22cf6d2f-4b51-4122-b1e1-f29aaccba528\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b966f085b0e7928104a487ee518ffabad37bfe2a5af66e9a41b5fedc221bbee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31441, \"end_char_idx\": 35553, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.\n    \n    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.\n    \n    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.\n    \n    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.\n    \n    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.\n    \n    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 22cf6d2f-4b51-4122-b1e1-f29aaccba528
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"22cf6d2f-4b51-4122-b1e1-f29aaccba528\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"009e71b1-981f-4fa5-b760-c4aa5a131c90\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"23a396ebac71e73fd5667caaff1fd4f2bec1560b0ecb0f683bb3f2dac7e0ff2a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"11926555-d728-4c09-9c7f-e627d2a1f1f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6dbd59c07ad24478c8d76be0d53bb8e72a00c6f9b5224dea60807120f226654d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33488, \"end_char_idx\": 37584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.\n    \n    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.\n    \n    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.\n    \n    The concept of \"Friendly AI\" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.\n    \n    Other approaches to AI ethics include Wendell Wallach's concept of \"artificial moral agents\" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.\n    \n    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.\n    \n    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 11926555-d728-4c09-9c7f-e627d2a1f1f1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"11926555-d728-4c09-9c7f-e627d2a1f1f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"22cf6d2f-4b51-4122-b1e1-f29aaccba528\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c59eca7ae72f7b566c24c3cb86b377e227ee7c83df822cefd54c90ea90fbf16f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c3e7efa5-a35e-4782-8841-3bb0ddf4db05\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"11840109df54a1b0d934e0054aaa8d15461fc64385f52be2a7382fa7ce8cd065\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 35491, \"end_char_idx\": 39409, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.\n    \n    The concept of \"Friendly AI\" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.\n    \n    Other approaches to AI ethics include Wendell Wallach's concept of \"artificial moral agents\" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.\n    \n    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.\n    \n    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.\n    \n    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.\n    \n    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.\n    \n    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.\n    \n    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an \"electronic brain.\"\n    \n    Early research in artificial intelligence (AI) included the development of \"artificial neurons\" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c3e7efa5-a35e-4782-8841-3bb0ddf4db05
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c3e7efa5-a35e-4782-8841-3bb0ddf4db05\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"11926555-d728-4c09-9c7f-e627d2a1f1f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"59ab7471190a82b9586d59abe2fb1c194d710a582e0323d2c686b661ba2e7046\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a573034e26b7f0acda81e701e0c415f8e5577ee3a3e300f41ce293816f815a69\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37475, \"end_char_idx\": 41372, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.\n    \n    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.\n    \n    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.\n    \n    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.\n    \n    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an \"electronic brain.\"\n    \n    Early research in artificial intelligence (AI) included the development of \"artificial neurons\" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence. The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.\n    \n    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the \"AI winter\" where funding was scarce.\n    \n    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.\n    \n    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c3e7efa5-a35e-4782-8841-3bb0ddf4db05\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2c4b611156ecb41838a6edc8be145e62ef91bbb675f22c1a8c0fdfccf3fb69fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7379fe50-66b1-4f1b-8a0d-a2c2a84bc622\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"13f2c9a2c3e9446172b0a79a8a36d81a9cc9f6f21555340cdf7b9c121750b821\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39410, \"end_char_idx\": 43068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.\n    \n    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the \"AI winter\" where funding was scarce.\n    \n    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.\n    \n    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits.\n    \n    AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.\n    \n    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.\n    \n    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.\n    \n    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7379fe50-66b1-4f1b-8a0d-a2c2a84bc622
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7379fe50-66b1-4f1b-8a0d-a2c2a84bc622\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b53fc5cf-d6d8-4921-b3d0-bdf1b53e95ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0b799772952f1e90ecb951eee3f88623fb22464fe203e5a0e1d6c7c208bbc8bb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aaa5acd2-cfc6-4ef3-8913-70466af670a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"864ec1be43f33832f22eb87518769abe3da151388b4c9d94758e1ddab55af094\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41382, \"end_char_idx\": 45201, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.\n    \n    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.\n    \n    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.\n    \n    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved. Alan Turing, in 1950, suggested shifting the focus from whether machines can think to whether they can exhibit intelligent behavior, as demonstrated by his Turing test, which assesses a machine's ability to simulate human conversation. Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.\n    \n    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.\n    \n    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.\n    \n    Symbolic AI, or \"GOFAI,\" focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.\n    \n    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aaa5acd2-cfc6-4ef3-8913-70466af670a1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aaa5acd2-cfc6-4ef3-8913-70466af670a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7379fe50-66b1-4f1b-8a0d-a2c2a84bc622\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"813b60ab99168799a13ae97f607b1b5200b5bc4d9d1e288056dcc3eab247b42e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"582e3747-b7b8-42bc-a1f2-09a41a42fea0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"581edbf37bf65be3eca384998eb42a7a7b72f7e2bdb5182dc8ec362f5d914196\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 43305, \"end_char_idx\": 47330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.\n    \n    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.\n    \n    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.\n    \n    Symbolic AI, or \"GOFAI,\" focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.\n    \n    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes. This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.\n    \n    In AI development, there has been a historical division between \"Neats,\" who believe intelligent behavior can be described with simple principles, and \"Scruffies,\" who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.\n    \n    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.\n    \n    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.\n    \n    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.\n    \n    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the \"hard\" problem of consciousness, which is understanding why or how brain processes feel like something, and the \"easy\" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 582e3747-b7b8-42bc-a1f2-09a41a42fea0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"582e3747-b7b8-42bc-a1f2-09a41a42fea0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aaa5acd2-cfc6-4ef3-8913-70466af670a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"efb5a6dca58996ed87446b03e31ade84f0a1d5c8dbe1afc8a110e54d4fe0b087\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"66ad818f-443e-4a43-beb3-a6962e4a228f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"398774b8edc77695f25fa8e430e5a28985bf15c9744e4117a0857105aae65ca5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 45202, \"end_char_idx\": 49352, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.\n    \n    In AI development, there has been a historical division between \"Neats,\" who believe intelligent behavior can be described with simple principles, and \"Scruffies,\" who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.\n    \n    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.\n    \n    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.\n    \n    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.\n    \n    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the \"hard\" problem of consciousness, which is understanding why or how brain processes feel like something, and the \"easy\" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain.\n    \n    In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.\n    \n    The concept of \"strong AI,\" as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting \"electronic personhood\" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.\n    \n    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.\n    \n    The concept of an \"intelligence explosion\" or \"singularity\" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 66ad818f-443e-4a43-beb3-a6962e4a228f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"66ad818f-443e-4a43-beb3-a6962e4a228f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"582e3747-b7b8-42bc-a1f2-09a41a42fea0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ad2f637a8d75ab1bb12403da5652de7b91aab45754e3ea9a6c1811d755e1753c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0d89c9cd-fff4-4dd5-8188-f425ed8fb30a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"735202f5d0d163c480984ba0dbbac7522e5875733118a19e29b5863de67afa79\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 47340, \"end_char_idx\": 51070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.\n    \n    The concept of \"strong AI,\" as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting \"electronic personhood\" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.\n    \n    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.\n    \n    The concept of an \"intelligence explosion\" or \"singularity\" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.\n    \n    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term \"robot\" was first introduced by Karel \u010capek in 1921. Notable narratives include Mary Shelley's \"Frankenstein\" and films like \"2001: A Space Odyssey\" and \"The Terminator,\" which typically showcase AI as a threat. Conversely, loyal robots like Gort from \"The Day the Earth Stood Still\" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.\n    \n    Numerous works, including Karel \u010capek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.\n\n\nNote that this utility also allows passing additional instructions.\n\n\n```python\nsummary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,\n                                                 additional_instructions=\"Write in point form and focus on numerical data.\")\nprint(summary_with_additional_instructions)\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [00:38<00:00,  7.73s/it]\n\n    - AI is intelligence demonstrated by machines, especially computer systems."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0d89c9cd-fff4-4dd5-8188-f425ed8fb30a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0d89c9cd-fff4-4dd5-8188-f425ed8fb30a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"66ad818f-443e-4a43-beb3-a6962e4a228f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"08903f2e07e2ab2705125126d62572c6d3a1fd6be6cacf5d945af6cec12ac066\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"85c9225d-3fe5-47ab-ace4-d24236caa327\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b6f7cd9a363899443e8bb9fede99ebb1981ee51391e798233b91d5e0d676868c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 49268, \"end_char_idx\": 53081, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.\n    \n    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term \"robot\" was first introduced by Karel \u010capek in 1921. Notable narratives include Mary Shelley's \"Frankenstein\" and films like \"2001: A Space Odyssey\" and \"The Terminator,\" which typically showcase AI as a threat. Conversely, loyal robots like Gort from \"The Day the Earth Stood Still\" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.\n    \n    Numerous works, including Karel \u010capek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.\n\n\nNote that this utility also allows passing additional instructions.\n\n\n```python\nsummary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,\n                                                 additional_instructions=\"Write in point form and focus on numerical data.\")\nprint(summary_with_additional_instructions)\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [00:38<00:00,  7.73s/it]\n\n    - AI is intelligence demonstrated by machines, especially computer systems.\n    - AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.\n    - Alan Turing initiated substantial AI research, termed \"machine intelligence.\"\n    - AI became an academic discipline in 1956, experiencing cycles of optimism and \"AI winters.\"\n    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.\n    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.\n    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.\n    - AI techniques include search, optimization, logic, neural networks, and statistical methods.\n    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.\n    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.\n    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.\n    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.\n    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.\n    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.\n    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.\n    - General AI aims to solve diverse problems with human-like versatility.\n    - AI search techniques include state space search, local search, and adversarial search for game-playing.\n    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 85c9225d-3fe5-47ab-ace4-d24236caa327
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"85c9225d-3fe5-47ab-ace4-d24236caa327\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0d89c9cd-fff4-4dd5-8188-f425ed8fb30a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5453c42c31d0447eaa16744b9861eedf55beb20ef035c244cf1281da5f823f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e4d40f8e-34de-4021-8e92-5603bd674c4b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2251f73f36332b9b5dd7e614be2e7eea5e64ccfe22a834d835dbe0496f2b2e33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 51075, \"end_char_idx\": 55070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "- AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.\n    - Alan Turing initiated substantial AI research, termed \"machine intelligence.\"\n    - AI became an academic discipline in 1956, experiencing cycles of optimism and \"AI winters.\"\n    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.\n    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.\n    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.\n    - AI techniques include search, optimization, logic, neural networks, and statistical methods.\n    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.\n    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.\n    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.\n    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.\n    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.\n    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.\n    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.\n    - General AI aims to solve diverse problems with human-like versatility.\n    - AI search techniques include state space search, local search, and adversarial search for game-playing.\n    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation.\n    - Probabilistic methods in AI address decision-making and planning under uncertainty using tools like Bayesian networks and Markov decision processes.\n    - Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.\n    \n    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.\n    - Deep neural networks: At least 2 hidden layers.\n    - Training techniques: Commonly use backpropagation.\n    - Feedforward networks: Signal passes in one direction.\n    - Recurrent networks: Output fed back into input for short-term memory.\n    - Perceptrons: Single layer of neurons.\n    - Convolutional networks: Strengthen connections between close neurons, important in image processing.\n    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.\n    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.\n    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.\n    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.\n    - AI in healthcare: Increases patient care, used in medical research and drug discovery.\n    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.\n    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.\n    - Generative AI: Creates realistic images and texts, used in creative arts.\n    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.\n    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.\n    \n    - AI engineers demographics: 4% black, 20% women.\n    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e4d40f8e-34de-4021-8e92-5603bd674c4b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e4d40f8e-34de-4021-8e92-5603bd674c4b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"85c9225d-3fe5-47ab-ace4-d24236caa327\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7acafa82a61d335e6c39aa2ff693c54d5a018ac485002455800a1c6ddea20e1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6e73abe5-4112-4981-be1a-87f7c19b47e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"722d5d71b192396b7060fdd300c835f901960f6e116f43af0787b9b06cb0c5df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 53241, \"end_char_idx\": 56938, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "- Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.\n    \n    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.\n    - Deep neural networks: At least 2 hidden layers.\n    - Training techniques: Commonly use backpropagation.\n    - Feedforward networks: Signal passes in one direction.\n    - Recurrent networks: Output fed back into input for short-term memory.\n    - Perceptrons: Single layer of neurons.\n    - Convolutional networks: Strengthen connections between close neurons, important in image processing.\n    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.\n    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.\n    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.\n    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.\n    - AI in healthcare: Increases patient care, used in medical research and drug discovery.\n    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.\n    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.\n    - Generative AI: Creates realistic images and texts, used in creative arts.\n    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.\n    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.\n    \n    - AI engineers demographics: 4% black, 20% women.\n    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias.\n    - AI complexity: Designers often can't explain decision-making processes.\n    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as \"cancerous\"; AI misclassifies asthma patients as low risk for pneumonia.\n    - Right to explanation: Essential for accountability, especially in medical and legal fields.\n    - DARPA's XAI program (2014): Aims to make AI decisions understandable.\n    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.\n    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.\n    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.\n    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.\n    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.\n    - Ethical AI development: Importance of aligning AI with human values and ethics.\n    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.\n    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.\n    \n    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.\n    - 1985: AI market value exceeded $1 billion.\n    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.\n    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.\n    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.\n    - 2012: Deep learning began dominating AI benchmarks.\n    - 2015-2019: Machine learning research publications increased by 50%."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6e73abe5-4112-4981-be1a-87f7c19b47e7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6e73abe5-4112-4981-be1a-87f7c19b47e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e4d40f8e-34de-4021-8e92-5603bd674c4b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"97b4da39108479dbdaccdc74ea4140b45cf8f555a10f856b99977317d20270fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4836df9d040963cf0e16b9da90dd973ca629a7122f0bc4f12d13376023401dd0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 55075, \"end_char_idx\": 58470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "- AI complexity: Designers often can't explain decision-making processes.\n    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as \"cancerous\"; AI misclassifies asthma patients as low risk for pneumonia.\n    - Right to explanation: Essential for accountability, especially in medical and legal fields.\n    - DARPA's XAI program (2014): Aims to make AI decisions understandable.\n    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.\n    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.\n    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.\n    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.\n    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.\n    - Ethical AI development: Importance of aligning AI with human values and ethics.\n    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.\n    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.\n    \n    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.\n    - 1985: AI market value exceeded $1 billion.\n    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.\n    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.\n    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.\n    - 2012: Deep learning began dominating AI benchmarks.\n    - 2015-2019: Machine learning research publications increased by 50%.\n    - 2016: Fairness and misuse of technology became central issues in AI.\n    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.\n    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.\n    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.\n    - 2010s: Statistical machine learning overshadowed other AI approaches.\n    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.\n    - Late 1980s: Introduction of soft computing techniques.\n    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).\n    - 2017: EU considered granting \"electronic personhood\" to advanced AI systems.\n    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.\n    \n    - Focus on how AI and technology, as depicted in \"Ex Machina\" and Philip K. Dick's \"Do Androids Dream of Electric Sheep?\", alter human subjectivity.\n    - No specific numerical data provided.\n\n\n    \n\n\nFinally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6e73abe5-4112-4981-be1a-87f7c19b47e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1272d25b95df50690896a0bbe6b7e2467661491b69e238e0a00527c51bb69679\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"622fb7e0-a21f-4cf4-9bae-4250afd14342\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6af23db3cfada4837d1c9eda7bfb3cfeb5b39b6719ced843d38bf788adb5e193\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 56596, \"end_char_idx\": 60318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "- 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.\n    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.\n    - 2012: Deep learning began dominating AI benchmarks.\n    - 2015-2019: Machine learning research publications increased by 50%.\n    - 2016: Fairness and misuse of technology became central issues in AI.\n    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.\n    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.\n    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.\n    - 2010s: Statistical machine learning overshadowed other AI approaches.\n    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.\n    - Late 1980s: Introduction of soft computing techniques.\n    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).\n    - 2017: EU considered granting \"electronic personhood\" to advanced AI systems.\n    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.\n    \n    - Focus on how AI and technology, as depicted in \"Ex Machina\" and Philip K. Dick's \"Do Androids Dream of Electric Sheep?\", alter human subjectivity.\n    - No specific numerical data provided.\n\n\n    \n\n\nFinally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary.\n\n\n```python\nrecursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)\nprint(recursive_summary)\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [00:41<00:00,  8.36s/it]\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as \"AI winters.\" Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.\n    \n    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.\n    \n    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.\n    \n    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 622fb7e0-a21f-4cf4-9bae-4250afd14342
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"622fb7e0-a21f-4cf4-9bae-4250afd14342\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7acc6db-d4b9-4f6e-b6cd-7fc34e0ee347\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"28f6cf44b6dfc6e07ceb7ca021cf307a49a6a4aedacb7898221b3e1685e78f18\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"04a4bc23-fa70-4f99-82df-88c3eb6b6cca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83289f7789368dd6ba4ebb66db50f7173e758ea8f9bd7d035c073e370feecc61\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 58473, \"end_char_idx\": 62711, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "```python\nrecursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)\nprint(recursive_summary)\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [00:41<00:00,  8.36s/it]\n\n    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as \"AI winters.\" Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.\n    \n    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.\n    \n    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.\n    \n    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles. AI research, which began as a distinct academic discipline in 1956, includes sub-fields like natural language processing and robotics, employing techniques from various scientific domains. AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.\n    \n    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.\n    \n    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.\n    \n    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as \"AI winters.\" Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.\n    \n    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.\n    \n    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 04a4bc23-fa70-4f99-82df-88c3eb6b6cca
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"04a4bc23-fa70-4f99-82df-88c3eb6b6cca\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"622fb7e0-a21f-4cf4-9bae-4250afd14342\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d23431ff39f20557d75f09d4eea04ad762c6ba893c72bf4034627c1967185577\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"538a5c65-1597-4d68-b0d9-247e559afe00\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9a866023518cbeefe2618e8ba1ba996eacd2f57fe8a01ef1a660743355cab8a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 60508, \"end_char_idx\": 64818, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.\n    \n    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.\n    \n    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.\n    \n    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as \"AI winters.\" Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.\n    \n    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.\n    \n    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks. AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.\n    \n    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.\n    \n    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first \"AI winter.\" The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring \"sub-symbolic\" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun\u2019s convolutional neural networks for digit recognition.\n    \n    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.\n    \n    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 538a5c65-1597-4d68-b0d9-247e559afe00
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"538a5c65-1597-4d68-b0d9-247e559afe00\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"dde4bb1f-6c81-4ea7-a2cc-407a65174124\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"04a4bc23-fa70-4f99-82df-88c3eb6b6cca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb\", \"title\": \"Summarizing Long Documents\", \"tokens\": 12631, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"941546bc1b255a66a985801db6cf8b82f90e33d9fea6315f187d0e3a6f84f5ae\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 62712, \"end_char_idx\": 65489, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "ref_doc_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "chroma:document": "AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.\n    \n    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.\n    \n    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first \"AI winter.\" The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring \"sub-symbolic\" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun\u2019s convolutional neural networks for digit recognition.\n    \n    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.\n    \n    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI. The concept of AI rights and welfare has also emerged, reflecting concerns about the moral status of advanced AI systems.\n    \n    Overall, AI research has oscillated between periods of intense optimism and profound setbacks, with current trends heavily favoring practical applications through narrow AI, while continuing to explore the broader implications and potential of general and superintelligent AI systems.\n    \n    Artificial Intelligence (AI) and its portrayal in media, such as the film \"Ex Machina\" and Philip K. Dick's novel \"Do Androids Dream of Electric Sheep?\", explore how technology, particularly AI, can alter our understanding of human subjectivity."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1e8ebf8d-d95c-4f15-9feb-5c4729395286
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1e8ebf8d-d95c-4f15-9feb-5c4729395286\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c298d6f-a68c-4f31-8644-06aaec1e0b1c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"06cc2ffe4a15ae4a93be1e8fe4717b9365c963fe9d70d2f2824f74c62e5c1b13\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "# How to use functions with a knowledge base\n\nThis notebook builds on the concepts in the [argument generation](How_to_call_functions_with_chat_models.ipynb) notebook, by creating an agent with access to a knowledge base and two functions that it can call based on the user requirement.\n\nWe'll create an agent that uses data from arXiv to answer questions about academic subjects. It has two functions at its disposal:\n- **get_articles**: A function that gets arXiv articles on a subject and summarizes them for the user with links.\n- **read_article_and_summarize**: This function takes one of the previously searched articles, reads it in its entirety and summarizes the core argument, evidence and conclusions.\n\nThis will get you comfortable with a multi-function workflow that can choose from multiple services, and where some of the data from the first function is persisted to be used by the second.\n\n## Walkthrough\n\nThis cookbook takes you through the following workflow:\n\n- **Search utilities:** Creating the two functions that access arXiv for answers.\n- **Configure Agent:** Building up the Agent behaviour that will assess the need for a function and, if one is required, call that function and present results back to the agent.\n- **arXiv conversation:** Put all of this together in live conversation.\n\n\n\n```python\n!pip install scipy --quiet\n!pip install tenacity --quiet\n!pip install tiktoken==0.3.3 --quiet\n!pip install termcolor --quiet\n!pip install openai --quiet\n!pip install arxiv --quiet\n!pip install pandas --quiet\n!pip install PyPDF2 --quiet\n!pip install tqdm --quiet\n```\n\n\n```python\nimport os\nimport arxiv\nimport ast\nimport concurrent\nimport json\nimport os\nimport pandas as pd\nimport tiktoken\nfrom csv import writer\nfrom IPython.display import display, Markdown, Latex\nfrom openai import OpenAI\nfrom PyPDF2 import PdfReader\nfrom scipy import spatial\nfrom tenacity import retry, wait_random_exponential, stop_after_attempt\nfrom tqdm import tqdm\nfrom termcolor import colored\n\nGPT_MODEL = \"gpt-3.5-turbo-0613\"\nEMBEDDING_MODEL = \"text-embedding-ada-002\"\nclient = OpenAI()\n```\n\n## Search utilities\n\nWe'll first set up some utilities that will underpin our two functions.\n\nDownloaded papers will be stored in a directory (we use ```./data/papers``` here). We create a file ```arxiv_library.csv``` to store the embeddings and details for downloaded papers to retrieve against using ```summarize_text```.\n\n\n```python\ndirectory = './data/papers'\n\n# Check if the directory already exists\nif not os.path.exists(directory):\n    # If the directory doesn't exist, create it and any necessary intermediate directories\n    os.makedirs(directory)\n    print(f\"Directory '{directory}' created successfully.\")\nelse:\n    # If the directory already exists, print a message indicating it\n    print(f\"Directory '{directory}' already exists.\")\n```\n\n    Directory './data/papers' already exists.\n\n\n\n```python\n# Set a directory to store downloaded papers\ndata_dir = os.path.join(os.curdir, \"data\", \"papers\")\npaper_dir_filepath = \"./data/arxiv_library.csv\"\n\n# Generate a blank dataframe where we can store downloaded files\ndf = pd.DataFrame(list())\ndf.to_csv(paper_dir_filepath)\n```\n\n\n```python\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef embedding_request(text):\n    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)\n    return response"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7c298d6f-a68c-4f31-8644-06aaec1e0b1c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7c298d6f-a68c-4f31-8644-06aaec1e0b1c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1e8ebf8d-d95c-4f15-9feb-5c4729395286\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"faf67a18fc144711c997eb09be7de41baccd8408d0ca6300cc64b34e87a1c239\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd1ea527-b4c9-4ead-861d-8cada290f9c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eeecb8750b86446247f52017531265dcf093443dfeec836b108fd5c697591a7a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2421, \"end_char_idx\": 5894, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\ndirectory = './data/papers'\n\n# Check if the directory already exists\nif not os.path.exists(directory):\n    # If the directory doesn't exist, create it and any necessary intermediate directories\n    os.makedirs(directory)\n    print(f\"Directory '{directory}' created successfully.\")\nelse:\n    # If the directory already exists, print a message indicating it\n    print(f\"Directory '{directory}' already exists.\")\n```\n\n    Directory './data/papers' already exists.\n\n\n\n```python\n# Set a directory to store downloaded papers\ndata_dir = os.path.join(os.curdir, \"data\", \"papers\")\npaper_dir_filepath = \"./data/arxiv_library.csv\"\n\n# Generate a blank dataframe where we can store downloaded files\ndf = pd.DataFrame(list())\ndf.to_csv(paper_dir_filepath)\n```\n\n\n```python\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef embedding_request(text):\n    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)\n    return response\n\n\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef get_articles(query, library=paper_dir_filepath, top_k=5):\n    \"\"\"This function gets the top_k articles based on a user's query, sorted by relevance.\n    It also downloads the files and stores them in arxiv_library.csv to be retrieved by the read_article_and_summarize.\n    \"\"\"\n    client = arxiv.Client()\n    search = arxiv.Search(\n        query = \"quantum\",\n        max_results = 10,\n        sort_by = arxiv.SortCriterion.SubmittedDate\n    )\n    result_list = []\n    for result in client.results(search):\n        result_dict = {}\n        result_dict.update({\"title\": result.title})\n        result_dict.update({\"summary\": result.summary})\n\n        # Taking the first url provided\n        result_dict.update({\"article_url\": [x.href for x in result.links][0]})\n        result_dict.update({\"pdf_url\": [x.href for x in result.links][1]})\n        result_list.append(result_dict)\n\n        # Store references in library file\n        response = embedding_request(text=result.title)\n        file_reference = [\n            result.title,\n            result.download_pdf(data_dir),\n            response.data[0].embedding,\n        ]\n\n        # Write to file\n        with open(library, \"a\") as f_object:\n            writer_object = writer(f_object)\n            writer_object.writerow(file_reference)\n            f_object.close()\n    return result_list\n\n```\n\n\n```python\n# Test that the search is working\nresult_output = get_articles(\"ppo reinforcement learning\")\nresult_output[0]\n\n```\n\n\n\n\n    {'title': 'Quantum types: going beyond qubits and quantum gates',\n     'summary': 'Quantum computing is a growing field with significant potential applications.\\nLearning how to code quantum programs means understanding how qubits work and\\nlearning to use quantum gates. This is analogous to creating classical\\nalgorithms using logic gates and bits. Even after learning all concepts, it is\\ndifficult to create new algorithms, which hinders the acceptance of quantum\\nprogramming by most developers. This article outlines the need for higher-level\\nabstractions and proposes some of them in a developer-friendly programming\\nlanguage called Rhyme. The new quantum types are extensions of classical types,\\nincluding bits, integers, floats, characters, arrays, and strings. We show how\\nto use such types with code snippets.',\n     'article_url': 'http://arxiv.org/abs/2401.15073v1',\n     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bd1ea527-b4c9-4ead-861d-8cada290f9c7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bd1ea527-b4c9-4ead-861d-8cada290f9c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c298d6f-a68c-4f31-8644-06aaec1e0b1c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7e02496287c4d245cfb6cc65fb2bf54b867c6d86561811fe3755f11da758fb3a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c103e97-4cf9-4391-a550-cf323ca1ec93\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7163b04905084568652b4d4061f5447108bd80f5e5f144bcc18056e7382f7dcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4824, \"end_char_idx\": 7822, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\n# Test that the search is working\nresult_output = get_articles(\"ppo reinforcement learning\")\nresult_output[0]\n\n```\n\n\n\n\n    {'title': 'Quantum types: going beyond qubits and quantum gates',\n     'summary': 'Quantum computing is a growing field with significant potential applications.\\nLearning how to code quantum programs means understanding how qubits work and\\nlearning to use quantum gates. This is analogous to creating classical\\nalgorithms using logic gates and bits. Even after learning all concepts, it is\\ndifficult to create new algorithms, which hinders the acceptance of quantum\\nprogramming by most developers. This article outlines the need for higher-level\\nabstractions and proposes some of them in a developer-friendly programming\\nlanguage called Rhyme. The new quantum types are extensions of classical types,\\nincluding bits, integers, floats, characters, arrays, and strings. We show how\\nto use such types with code snippets.',\n     'article_url': 'http://arxiv.org/abs/2401.15073v1',\n     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}\n\n\n\n\n```python\ndef strings_ranked_by_relatedness(\n    query: str,\n    df: pd.DataFrame,\n    relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),\n    top_n: int = 100,\n) -> list[str]:\n    \"\"\"Returns a list of strings and relatednesses, sorted from most related to least.\"\"\"\n    query_embedding_response = embedding_request(query)\n    query_embedding = query_embedding_response.data[0].embedding\n    strings_and_relatednesses = [\n        (row[\"filepath\"], relatedness_fn(query_embedding, row[\"embedding\"]))\n        for i, row in df.iterrows()\n    ]\n    strings_and_relatednesses.sort(key=lambda x: x[1], reverse=True)\n    strings, relatednesses = zip(*strings_and_relatednesses)\n    return strings[:top_n]\n\n```\n\n\n```python\ndef read_pdf(filepath):\n    \"\"\"Takes a filepath to a PDF and returns a string of the PDF's contents\"\"\"\n    # creating a pdf reader object\n    reader = PdfReader(filepath)\n    pdf_text = \"\"\n    page_number = 0\n    for page in reader.pages:\n        page_number += 1\n        pdf_text += page.extract_text() + f\"\\nPage Number: {page_number}\"\n    return pdf_text\n\n\n# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\ndef create_chunks(text, n, tokenizer):\n    \"\"\"Returns successive n-sized chunks from provided text.\"\"\"\n    tokens = tokenizer.encode(text)\n    i = 0\n    while i < len(tokens):\n        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n        j = min(i + int(1.5 * n), len(tokens))\n        while j > i + int(0.5 * n):\n            # Decode the tokens and check for full stop or newline\n            chunk = tokenizer.decode(tokens[i:j])\n            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n                break\n            j -= 1\n        # If no end of sentence found, use n tokens as the chunk size\n        if j == i + int(0.5 * n):\n            j = min(i + n, len(tokens))\n        yield tokens[i:j]\n        i = j"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6c103e97-4cf9-4391-a550-cf323ca1ec93
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6c103e97-4cf9-4391-a550-cf323ca1ec93\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd1ea527-b4c9-4ead-861d-8cada290f9c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"66ebd595220fcf8427bec5345ba9961fb2f8eba209bf2a68f1846be54f97c071\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c21350d0-23de-41c9-90d7-5ef0f09dc14b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8368b1e892457b8dadde4755ca24de7594d075628542d9c89386fa85104d0de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6615, \"end_char_idx\": 8210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\ndef read_pdf(filepath):\n    \"\"\"Takes a filepath to a PDF and returns a string of the PDF's contents\"\"\"\n    # creating a pdf reader object\n    reader = PdfReader(filepath)\n    pdf_text = \"\"\n    page_number = 0\n    for page in reader.pages:\n        page_number += 1\n        pdf_text += page.extract_text() + f\"\\nPage Number: {page_number}\"\n    return pdf_text\n\n\n# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\ndef create_chunks(text, n, tokenizer):\n    \"\"\"Returns successive n-sized chunks from provided text.\"\"\"\n    tokens = tokenizer.encode(text)\n    i = 0\n    while i < len(tokens):\n        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n        j = min(i + int(1.5 * n), len(tokens))\n        while j > i + int(0.5 * n):\n            # Decode the tokens and check for full stop or newline\n            chunk = tokenizer.decode(tokens[i:j])\n            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n                break\n            j -= 1\n        # If no end of sentence found, use n tokens as the chunk size\n        if j == i + int(0.5 * n):\n            j = min(i + n, len(tokens))\n        yield tokens[i:j]\n        i = j\n\n\ndef extract_chunk(content, template_prompt):\n    \"\"\"This function applies a prompt to some input content. In this case it returns a summarized chunk of text\"\"\"\n    prompt = template_prompt + content\n    response = client.chat.completions.create(\n        model=GPT_MODEL, messages=[{\"role\": \"user\", \"content\": prompt}], temperature=0\n    )\n    return response.choices[0].message.content"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c21350d0-23de-41c9-90d7-5ef0f09dc14b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c21350d0-23de-41c9-90d7-5ef0f09dc14b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c103e97-4cf9-4391-a550-cf323ca1ec93\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"82e2368b97ff83bebf2e1332c096754ed5324a9487ccdcc17b4c29d006727895\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"631a23d4-4e44-41d6-aca0-40f0b3af43c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f17afc756f7302089437434ee8c74203da2c741dae57a6945a3895b22db4707\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6615, \"end_char_idx\": 11063, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\ndef read_pdf(filepath):\n    \"\"\"Takes a filepath to a PDF and returns a string of the PDF's contents\"\"\"\n    # creating a pdf reader object\n    reader = PdfReader(filepath)\n    pdf_text = \"\"\n    page_number = 0\n    for page in reader.pages:\n        page_number += 1\n        pdf_text += page.extract_text() + f\"\\nPage Number: {page_number}\"\n    return pdf_text\n\n\n# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\ndef create_chunks(text, n, tokenizer):\n    \"\"\"Returns successive n-sized chunks from provided text.\"\"\"\n    tokens = tokenizer.encode(text)\n    i = 0\n    while i < len(tokens):\n        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n        j = min(i + int(1.5 * n), len(tokens))\n        while j > i + int(0.5 * n):\n            # Decode the tokens and check for full stop or newline\n            chunk = tokenizer.decode(tokens[i:j])\n            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n                break\n            j -= 1\n        # If no end of sentence found, use n tokens as the chunk size\n        if j == i + int(0.5 * n):\n            j = min(i + n, len(tokens))\n        yield tokens[i:j]\n        i = j\n\n\ndef extract_chunk(content, template_prompt):\n    \"\"\"This function applies a prompt to some input content. In this case it returns a summarized chunk of text\"\"\"\n    prompt = template_prompt + content\n    response = client.chat.completions.create(\n        model=GPT_MODEL, messages=[{\"role\": \"user\", \"content\": prompt}], temperature=0\n    )\n    return response.choices[0].message.content\n\n\ndef summarize_text(query):\n    \"\"\"This function does the following:\n    - Reads in the arxiv_library.csv file in including the embeddings\n    - Finds the closest file to the user's query\n    - Scrapes the text out of the file and chunks it\n    - Summarizes each chunk in parallel\n    - Does one final summary and returns this to the user\"\"\"\n\n    # A prompt to dictate how the recursive summarizations should approach the input paper\n    summary_prompt = \"\"\"Summarize this text from an academic paper. Extract any key points with reasoning.\\n\\nContent:\"\"\"\n\n    # If the library is empty (no searches have been performed yet), we perform one and download the results\n    library_df = pd.read_csv(paper_dir_filepath).reset_index()\n    if len(library_df) == 0:\n        print(\"No papers searched yet, downloading first.\")\n        get_articles(query)\n        print(\"Papers downloaded, continuing\")\n        library_df = pd.read_csv(paper_dir_filepath).reset_index()\n    library_df.columns = [\"title\", \"filepath\", \"embedding\"]\n    library_df[\"embedding\"] = library_df[\"embedding\"].apply(ast.literal_eval)\n    strings = strings_ranked_by_relatedness(query, library_df, top_n=1)\n    print(\"Chunking text from paper\")\n    pdf_text = read_pdf(strings[0])\n\n    # Initialise tokenizer\n    tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n    results = \"\"\n\n    # Chunk up the document into 1500 token chunks\n    chunks = create_chunks(pdf_text, 1500, tokenizer)\n    text_chunks = [tokenizer.decode(chunk) for chunk in chunks]\n    print(\"Summarizing each chunk of text\")\n\n    # Parallel process the summaries\n    with concurrent.futures.ThreadPoolExecutor(\n        max_workers=len(text_chunks)\n    ) as executor:\n        futures = [\n            executor.submit(extract_chunk, chunk, summary_prompt)\n            for chunk in text_chunks\n        ]\n        with tqdm(total=len(text_chunks)) as pbar:\n            for _ in concurrent.futures.as_completed(futures):\n                pbar.update(1)\n        for future in futures:\n            data = future.result()\n            results += data\n\n    # Final summary\n    print(\"Summarizing into overall summary\")\n    response = client.chat.completions.create(\n        model=GPT_MODEL,\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": f\"\"\"Write a summary collated from this collection of key points extracted from an academic paper.\n                        The summary should highlight the core argument, conclusions and evidence, and answer the user's query.\n                        User query: {query}\n                        The summary should be structured in bulleted lists following the headings Core Argument, Evidence, and Conclusions.\n                        Key points:\\n{results}\\nSummary:\\n\"\"\",\n            }\n        ],\n        temperature=0,\n    )\n    return response\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 631a23d4-4e44-41d6-aca0-40f0b3af43c8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"631a23d4-4e44-41d6-aca0-40f0b3af43c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c21350d0-23de-41c9-90d7-5ef0f09dc14b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e3938444d746a767bd2eec1d57e73306a7398c1cb3385ff2aaba09225f3d5110\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"705c20f1-15f0-4c96-86f3-f391d529a791\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f972f6c387d124395a2a0fabb2b65591ec6d453ecca3ba9859f831cad21e7b0f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11066, \"end_char_idx\": 14574, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\n# Test the summarize_text function works\nchat_test_response = summarize_text(\"PPO reinforcement learning sequence generation\")\n\n```\n\n    Chunking text from paper\n    Summarizing each chunk of text\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 6/6 [00:06<00:00,  1.08s/it]\n\n\n    Summarizing into overall summary\n\n\n\n```python\nprint(chat_test_response.choices[0].message.content)\n\n```\n\n    Core Argument:\n    - The academic paper explores the connection between the transverse field Ising (TFI) model and the \u03d54 model, highlighting the analogy between topological solitary waves in the \u03d54 model and the effect of the transverse field on spin flips in the TFI model.\n    - The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical \u03d54 model subjected to periodic perturbations, which are essential in annealing phenomena.\n    - The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.\n    \n    Evidence:\n    - The authors analyze the dynamics and relaxation of weakly coupled \u03d54 chains through numerical simulations, observing kink and breather excitations and investigating the structural phase transition associated with the double well potential.\n    - The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C\u22a5) approaches zero, but there is a finite Tc for C\u22a5>0.\n    - The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.\n    - The soft-mode frequency (\u03c9s) decreases as temperature approaches Ts, the dynamical crossover temperature.\n    - The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.\n    - The mean domain size (\u27e8DS\u27e9) exhibits an algebraic decay for finite C\u22a5>0.\n    - The probability of larger domain sizes is higher before a kick compared to after a kick for C\u22a5>0.\n    \n    Conclusions:\n    - The authors suggest further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space.\n    - They propose extending the study to different kick profiles, introducing kink defects, and studying weakly-coupled chains in higher dimensions.\n    - Recognizing similarities between classical nonlinear equations and quantum linear ones in information processing is important.\n    - Future research directions include investigating the dynamics of quantum annealing, measurement and memory in the periodically driven complex Ginzburg-Landau equation, and the behavior of solitons and domain walls in various systems.\n\n\n## Configure Agent\n\nWe'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.\n\n\n```python\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef chat_completion_request(messages, functions=None, model=GPT_MODEL):\n    try:\n        response = client.chat.completions.create(\n            model=model,\n            messages=messages,\n            functions=functions,\n        )\n        return response\n    except Exception as e:\n        print(\"Unable to generate ChatCompletion response\")\n        print(f\"Exception: {e}\")\n        return e\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 705c20f1-15f0-4c96-86f3-f391d529a791
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"705c20f1-15f0-4c96-86f3-f391d529a791\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"631a23d4-4e44-41d6-aca0-40f0b3af43c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"202aceb1b7ba433661ff9a1d21ca08b4e64d6ac47dfae01ca3cf4a0c6773cd25\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a49b643b-e2d6-49c1-8a7e-b244ce45fb49\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f2c2739b500be4e6dd472eb17df76ed9fb7afeff7e2baf78e6697081e179d747\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13833, \"end_char_idx\": 17168, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "## Configure Agent\n\nWe'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.\n\n\n```python\n@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))\ndef chat_completion_request(messages, functions=None, model=GPT_MODEL):\n    try:\n        response = client.chat.completions.create(\n            model=model,\n            messages=messages,\n            functions=functions,\n        )\n        return response\n    except Exception as e:\n        print(\"Unable to generate ChatCompletion response\")\n        print(f\"Exception: {e}\")\n        return e\n\n```\n\n\n```python\nclass Conversation:\n    def __init__(self):\n        self.conversation_history = []\n\n    def add_message(self, role, content):\n        message = {\"role\": role, \"content\": content}\n        self.conversation_history.append(message)\n\n    def display_conversation(self, detailed=False):\n        role_to_color = {\n            \"system\": \"red\",\n            \"user\": \"green\",\n            \"assistant\": \"blue\",\n            \"function\": \"magenta\",\n        }\n        for message in self.conversation_history:\n            print(\n                colored(\n                    f\"{message['role']}: {message['content']}\\n\\n\",\n                    role_to_color[message[\"role\"]],\n                )\n            )\n```\n\n\n```python\n# Initiate our get_articles and read_article_and_summarize functions\narxiv_functions = [\n    {\n        \"name\": \"get_articles\",\n        \"description\": \"\"\"Use this function to get academic papers from arXiv to answer user questions.\"\"\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\n                    \"type\": \"string\",\n                    \"description\": f\"\"\"\n                            User query in JSON. Responses should be summarized and should include the article URL reference\n                            \"\"\",\n                }\n            },\n            \"required\": [\"query\"],\n        },\n    },\n    {\n        \"name\": \"read_article_and_summarize\",\n        \"description\": \"\"\"Use this function to read whole papers and provide a summary for users.\n        You should NEVER call this function before get_articles has been called in the conversation.\"\"\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\n                    \"type\": \"string\",\n                    \"description\": f\"\"\"\n                            Description of the article in plain text based on the user's query\n                            \"\"\",\n                }\n            },\n            \"required\": [\"query\"],\n        },\n    }\n]\n\n```\n\n\n```python\ndef chat_completion_with_function_execution(messages, functions=[None]):\n    \"\"\"This function makes a ChatCompletion API call with the option of adding functions\"\"\"\n    response = chat_completion_request(messages, functions)\n    full_message = response.choices[0]\n    if full_message.finish_reason == \"function_call\":\n        print(f\"Function generation requested, calling function\")\n        return call_arxiv_function(messages, full_message)\n    else:\n        print(f\"Function not required, responding to user\")\n        return response"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a49b643b-e2d6-49c1-8a7e-b244ce45fb49
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a49b643b-e2d6-49c1-8a7e-b244ce45fb49\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"705c20f1-15f0-4c96-86f3-f391d529a791\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d1388e240614fa5fac7593c9eb1972906f54a3cc155e7b603c76ec3cd6ff1bae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"87436c88-44b5-41a5-9065-b95cba2a29b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d17641a152018161c22c138deb19fbb3810445cb2e00f5e9e44e55ad7bb4c95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15283, \"end_char_idx\": 19327, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\n# Initiate our get_articles and read_article_and_summarize functions\narxiv_functions = [\n    {\n        \"name\": \"get_articles\",\n        \"description\": \"\"\"Use this function to get academic papers from arXiv to answer user questions.\"\"\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\n                    \"type\": \"string\",\n                    \"description\": f\"\"\"\n                            User query in JSON. Responses should be summarized and should include the article URL reference\n                            \"\"\",\n                }\n            },\n            \"required\": [\"query\"],\n        },\n    },\n    {\n        \"name\": \"read_article_and_summarize\",\n        \"description\": \"\"\"Use this function to read whole papers and provide a summary for users.\n        You should NEVER call this function before get_articles has been called in the conversation.\"\"\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\n                    \"type\": \"string\",\n                    \"description\": f\"\"\"\n                            Description of the article in plain text based on the user's query\n                            \"\"\",\n                }\n            },\n            \"required\": [\"query\"],\n        },\n    }\n]\n\n```\n\n\n```python\ndef chat_completion_with_function_execution(messages, functions=[None]):\n    \"\"\"This function makes a ChatCompletion API call with the option of adding functions\"\"\"\n    response = chat_completion_request(messages, functions)\n    full_message = response.choices[0]\n    if full_message.finish_reason == \"function_call\":\n        print(f\"Function generation requested, calling function\")\n        return call_arxiv_function(messages, full_message)\n    else:\n        print(f\"Function not required, responding to user\")\n        return response\n\n\ndef call_arxiv_function(messages, full_message):\n    \"\"\"Function calling function which executes function calls when the model believes it is necessary.\n    Currently extended by adding clauses to this if statement.\"\"\"\n\n    if full_message.message.function_call.name == \"get_articles\":\n        try:\n            parsed_output = json.loads(\n                full_message.message.function_call.arguments\n            )\n            print(\"Getting search results\")\n            results = get_articles(parsed_output[\"query\"])\n        except Exception as e:\n            print(parsed_output)\n            print(f\"Function execution failed\")\n            print(f\"Error message: {e}\")\n        messages.append(\n            {\n                \"role\": \"function\",\n                \"name\": full_message.message.function_call.name,\n                \"content\": str(results),\n            }\n        )\n        try:\n            print(\"Got search results, summarizing content\")\n            response = chat_completion_request(messages)\n            return response\n        except Exception as e:\n            print(type(e))\n            raise Exception(\"Function chat request failed\")\n\n    elif (\n        full_message.message.function_call.name == \"read_article_and_summarize\"\n    ):\n        parsed_output = json.loads(\n            full_message.message.function_call.arguments\n        )\n        print(\"Finding and reading paper\")\n        summary = summarize_text(parsed_output[\"query\"])\n        return summary\n\n    else:\n        raise Exception(\"Function does not exist and cannot be called\")\n\n```\n\n## arXiv conversation\n\nLet's put this all together by testing our functions out in conversation.\n\n\n```python\n# Start with a system message\npaper_system_message = \"\"\"You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.\nYou summarize the papers clearly so the customer can decide which to read to answer their question.\nYou always provide the article_url and title so the user can understand the name of the paper and click through to access it.\nBegin!\"\"\"\npaper_conversation = Conversation()\npaper_conversation.add_message(\"system\", paper_system_message)\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 87436c88-44b5-41a5-9065-b95cba2a29b6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"87436c88-44b5-41a5-9065-b95cba2a29b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a49b643b-e2d6-49c1-8a7e-b244ce45fb49\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"880985dc24aac144d7436f46d8a05fb03dbd7f87f8391ed9d30fe2d22b71e41c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0671465c-7c16-48da-a8f2-3c5ceb188b34\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"080f1c8dcfd97e8e4912db3e5715f056283b13c875b328da309ff87e2424e2cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17171, \"end_char_idx\": 19868, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "def call_arxiv_function(messages, full_message):\n    \"\"\"Function calling function which executes function calls when the model believes it is necessary.\n    Currently extended by adding clauses to this if statement.\"\"\"\n\n    if full_message.message.function_call.name == \"get_articles\":\n        try:\n            parsed_output = json.loads(\n                full_message.message.function_call.arguments\n            )\n            print(\"Getting search results\")\n            results = get_articles(parsed_output[\"query\"])\n        except Exception as e:\n            print(parsed_output)\n            print(f\"Function execution failed\")\n            print(f\"Error message: {e}\")\n        messages.append(\n            {\n                \"role\": \"function\",\n                \"name\": full_message.message.function_call.name,\n                \"content\": str(results),\n            }\n        )\n        try:\n            print(\"Got search results, summarizing content\")\n            response = chat_completion_request(messages)\n            return response\n        except Exception as e:\n            print(type(e))\n            raise Exception(\"Function chat request failed\")\n\n    elif (\n        full_message.message.function_call.name == \"read_article_and_summarize\"\n    ):\n        parsed_output = json.loads(\n            full_message.message.function_call.arguments\n        )\n        print(\"Finding and reading paper\")\n        summary = summarize_text(parsed_output[\"query\"])\n        return summary\n\n    else:\n        raise Exception(\"Function does not exist and cannot be called\")\n\n```\n\n## arXiv conversation\n\nLet's put this all together by testing our functions out in conversation.\n\n\n```python\n# Start with a system message\npaper_system_message = \"\"\"You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.\nYou summarize the papers clearly so the customer can decide which to read to answer their question.\nYou always provide the article_url and title so the user can understand the name of the paper and click through to access it.\nBegin!\"\"\"\npaper_conversation = Conversation()\npaper_conversation.add_message(\"system\", paper_system_message)\n\n```\n\n\n```python\n# Add a user message\npaper_conversation.add_message(\"user\", \"Hi, how does PPO reinforcement learning work?\")\nchat_response = chat_completion_with_function_execution(\n    paper_conversation.conversation_history, functions=arxiv_functions\n)\nassistant_message = chat_response.choices[0].message.content\npaper_conversation.add_message(\"assistant\", assistant_message)\ndisplay(Markdown(assistant_message))\n\n```\n\n    Function generation requested, calling function\n    Getting search results\n    Got search results, summarizing content"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0671465c-7c16-48da-a8f2-3c5ceb188b34
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0671465c-7c16-48da-a8f2-3c5ceb188b34\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"87436c88-44b5-41a5-9065-b95cba2a29b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1e746dd834dac2580d078e7ca05bb48e7acd2d9cfdb159c131ea106b9e45d11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1a6ae21f-bdb9-4a90-94d6-6d73f6b5ec0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"925f77bc44c7ace54858e6dfd2cb44cae9d97e6e9b6542607e6a17b25effd08c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18836, \"end_char_idx\": 22344, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\n# Start with a system message\npaper_system_message = \"\"\"You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.\nYou summarize the papers clearly so the customer can decide which to read to answer their question.\nYou always provide the article_url and title so the user can understand the name of the paper and click through to access it.\nBegin!\"\"\"\npaper_conversation = Conversation()\npaper_conversation.add_message(\"system\", paper_system_message)\n\n```\n\n\n```python\n# Add a user message\npaper_conversation.add_message(\"user\", \"Hi, how does PPO reinforcement learning work?\")\nchat_response = chat_completion_with_function_execution(\n    paper_conversation.conversation_history, functions=arxiv_functions\n)\nassistant_message = chat_response.choices[0].message.content\npaper_conversation.add_message(\"assistant\", assistant_message)\ndisplay(Markdown(assistant_message))\n\n```\n\n    Function generation requested, calling function\n    Getting search results\n    Got search results, summarizing content\n\n\n\nPPO (Proximal Policy Optimization) is a reinforcement learning algorithm that aims to find the optimal policy for an agent by optimizing the policy parameters in an iterative manner. Here are a few papers that discuss PPO in more detail:\n\n1. Title: \"Proximal Policy Optimization Algorithms\"\n   Article URL: [arxiv.org/abs/1707.06347v2](http://arxiv.org/abs/1707.06347v2)\n   Summary: This paper introduces two algorithms, PPO (Proximal Policy Optimization) and TRPO (Trust Region Policy Optimization), that address the issue of sample efficiency and stability in reinforcement learning. PPO uses a surrogate objective function that makes smaller updates to the policy parameters, resulting in more stable and efficient learning.\n\n2. Title: \"Emergence of Locomotion Behaviours in Rich Environments with PPO\"\n   Article URL: [arxiv.org/abs/1707.02286v3](http://arxiv.org/abs/1707.02286v3)\n   Summary: This paper explores the use of PPO in training agents to learn locomotion behaviors in complex and dynamic environments. The authors demonstrate the effectiveness of PPO in learning a variety of locomotion skills, such as walking, jumping, and climbing.\n\n3. Title: \"Proximal Policy Optimization for Multi-Agent Systems\"\n   Article URL: [arxiv.org/abs/2006.14171v2](http://arxiv.org/abs/2006.14171v2)\n   Summary: This paper extends PPO to the domain of multi-agent systems, where multiple agents interact and learn together. The authors propose a decentralized version of PPO that allows each agent to update its policy independently based on its local observations, resulting in more scalable and efficient learning in multi-agent environments.\n\nThese papers provide detailed explanations of the PPO algorithm, its advantages, and its applications in different scenarios. Reading them can give you a deeper understanding of how PPO reinforcement learning works.\n\n\n\n```python\n# Add another user message to induce our system to use the second tool\npaper_conversation.add_message(\n    \"user\",\n    \"Can you read the PPO sequence generation paper for me and give me a summary\",\n)\nupdated_response = chat_completion_with_function_execution(\n    paper_conversation.conversation_history, functions=arxiv_functions\n)\ndisplay(Markdown(updated_response.choices[0].message.content))\n\n```\n\n    Function generation requested, calling function\n    Finding and reading paper\n    Chunking text from paper\n    Summarizing each chunk of text\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 6/6 [00:07<00:00,  1.19s/it]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1a6ae21f-bdb9-4a90-94d6-6d73f6b5ec0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1a6ae21f-bdb9-4a90-94d6-6d73f6b5ec0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e1af5205-07a6-44b4-a2f6-def09084e159\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8cf1902188ba73e42003f2a65860aefc2a7aafdc7bb3741f4ae979e0e0199c72\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0671465c-7c16-48da-a8f2-3c5ceb188b34\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb\", \"title\": \"How to use functions with a knowledge base\", \"tokens\": 5216, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"467260934d65843d64863bfe80d7618ecb2ddeeac6279b8a65e899d4a7014647\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21735, \"end_char_idx\": 24713, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "ref_doc_id": "e1af5205-07a6-44b4-a2f6-def09084e159", "chroma:document": "```python\n# Add another user message to induce our system to use the second tool\npaper_conversation.add_message(\n    \"user\",\n    \"Can you read the PPO sequence generation paper for me and give me a summary\",\n)\nupdated_response = chat_completion_with_function_execution(\n    paper_conversation.conversation_history, functions=arxiv_functions\n)\ndisplay(Markdown(updated_response.choices[0].message.content))\n\n```\n\n    Function generation requested, calling function\n    Finding and reading paper\n    Chunking text from paper\n    Summarizing each chunk of text\n\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 6/6 [00:07<00:00,  1.19s/it]\n\n\n    Summarizing into overall summary\n\n\n\nCore Argument:\n- The academic paper explores the connection between the transverse field Ising (TFI) model and the \u03d54 model, highlighting the analogy between the coupling of topological solitary waves in the \u03d54 model and the effect of the transverse field on spin flips in the TFI model.\n- The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical \u03d54 model subjected to periodic perturbations, which are essential in annealing phenomena.\n- The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.\n\nEvidence:\n- The authors analyze the dynamics and relaxation of weakly coupled \u03d54 chains through numerical simulations, studying the behavior of kink and breather excitations and the structural phase transition associated with the double well potential.\n- The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C\u22a5) approaches zero, but there is a finite Tc for C\u22a5>0.\n- The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.\n- The soft-mode frequency (\u03c9s) decreases as temperature approaches Ts, the dynamical crossover temperature.\n- The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.\n- The mean domain size (\u27e8DS\u27e9) exhibits an algebraic decay for finite C\u22a5>0.\n- The probability of larger domain sizes is higher before a kick compared to after a kick for C\u22a5>0.\n\nConclusions:\n- The study of weakly-coupled classical \u03d54 chains provides insights into quantum annealing architectures and the role of topological excitations in these systems.\n- The equilibration of the system is faster for higher kick strengths, and the mean domain size increases with higher final temperatures.\n- Further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space is suggested.\n- The paper highlights the importance of recognizing similarities between classical nonlinear equations and quantum linear ones in information processing and suggests future research directions in this area."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5072734c-13c4-4ca6-9e19-ccd77a28dfaf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5072734c-13c4-4ca6-9e19-ccd77a28dfaf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54a9a06e-97ba-44f9-a854-5d10a9aed824\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"49ee01e928a34ae21f1de9c7e8e3725cbaab931199fb73255f1b385b51faa3eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2711, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "# Synthetic Data generation (Part 1)\n\n\nSynthetic data generation using large language models (LLMs) offers a powerful solution to a commonly faced problem: the availability of high-quality, diverse, and privacy-compliant data. This could be used in a number of scenarios such as training a data science  machine learning model (SVMs, decision trees, KNN's), finetuning a different GPT model on the data, as a solution to the coldstart problem, helping build compelling demos/apps with realistic data, scenario testing etc.\n\nThere are a number of key drivers which may see you wanting to leverage synthetic data. \n1. Human data may have privacy restrictions and/or identifiable data within it which we do not want to be used. \n2. Synthetic data can be much more structured and therefore easier to manipulate than real data. \n3. In domains where data is sparse or data of certain categories is sparse we may want to augment the data. \n4. When dealing with imbalanced datasets or datasets which lack diversity, we may want to create data to improve the richness of our datasets.\n\nUnlike traditional data augmentation or manual data creation methods, using LLMs allows for the generation of rich, nuanced, and contextually relevant datasets that can significantly enhance it's usefulness to enterprises and developers.\n\nWe split this tutorial into 2 parts. In this cookbook, we will have the following agenda:\n1. CSV with a structured prompt\n2. CSV with a Python program\n3. Multitable CSV with a python program\n4. Simply creating textual data\n5. Dealing with imbalanced or non-diverse textual data\nwhile in part 2, we will look at prompting strategies for getting better textual data.\n\nThe last two in particular are useful for creating synthetic data to finetune another GPT model. For example using higher quality data produced by `gpt-4o` to finetune the cheaper and quicker `gpt-3.5-turbo` for improved performance while reducing costs.\n\n\n### Getting setup\n\n\n```python\n%pip install openai\n%pip install pandas\n%pip install scikit-learn\n%pip install matplotlib\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport numpy as np\nimport pandas as pd\nfrom sklearn.cluster import KMeans\nimport matplotlib.pyplot as plt\nimport json\nimport matplotlib\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n```\n\n### 1. CSV with a structure prompt\nHere we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 54a9a06e-97ba-44f9-a854-5d10a9aed824
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"54a9a06e-97ba-44f9-a854-5d10a9aed824\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5072734c-13c4-4ca6-9e19-ccd77a28dfaf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1a10a5708848ffdc9b96923f94f67c7df9cf591a860b78e04dee0829e66042ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e26ce35e-be5f-4c81-864f-c20b71ac5f71\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2c7a757767a39d3c8af34856581764743b042af8eda1205092cc674d8f566f9d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1939, \"end_char_idx\": 4373, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "### Getting setup\n\n\n```python\n%pip install openai\n%pip install pandas\n%pip install scikit-learn\n%pip install matplotlib\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport numpy as np\nimport pandas as pd\nfrom sklearn.cluster import KMeans\nimport matplotlib.pyplot as plt\nimport json\nimport matplotlib\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n```\n\n### 1. CSV with a structure prompt\nHere we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance).\n\n\n```python\ndatagen_model = \"gpt-4o-mini\"\nquestion = \"\"\"\nCreate a CSV file with 10 rows of housing data.\nEach row should include the following fields:\n - id (incrementing integer starting at 1)\n - house size (m^2)\n - house price\n - location\n - number of bedrooms\n\nMake sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense). Also only respond with the CSV.\n\"\"\"\n\nresponse = client.chat.completions.create(\n  model=datagen_model,\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n    {\"role\": \"user\", \"content\": question}\n  ]\n)\nres = response.choices[0].message.content\nprint(res)\n```\n\n    ```csv\n    id,house_size_m2,house_price,location,number_of_bedrooms\n    1,50,150000,Suburban,2\n    2,75,250000,City Center,3\n    3,100,350000,Suburban,4\n    4,120,450000,Suburban,4\n    5,80,300000,City Center,3\n    6,90,400000,City Center,3\n    7,150,600000,Premium Area,5\n    8,200,750000,Premium Area,5\n    9,55,180000,Suburban,2\n    10,300,950000,Premium Area,6\n    ```\n\n\n### 2. CSV with a Python program\nThe issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.\n\nThis would then let us edit the python program as we desire while giving us a good basis to start from."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e26ce35e-be5f-4c81-864f-c20b71ac5f71
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e26ce35e-be5f-4c81-864f-c20b71ac5f71\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"54a9a06e-97ba-44f9-a854-5d10a9aed824\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9813953a1dd63f836586d838f9e966e66d49c5832896e589a9f66292182b1ff8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e4b3853-1cdb-495d-99e8-85d626030bde\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"aa076a8765b2bab84cd13af17a8842a34f01bbde49883af51ed15f092f67a5f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3878, \"end_char_idx\": 7670, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "### 2. CSV with a Python program\nThe issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.\n\nThis would then let us edit the python program as we desire while giving us a good basis to start from.\n\n\n\n```python\nquestion = \"\"\"\nCreate a Python program to generate 100 rows of housing data.\nI want you to at the end of it output a pandas dataframe with 100 rows of data.\nEach row should include the following fields:\n - id (incrementing integer starting at 1)\n - house size (m^2)\n - house price\n - location\n - number of bedrooms\n\nMake sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).\n\"\"\"\n\nresponse = client.chat.completions.create(\n  model=datagen_model,\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n    {\"role\": \"user\", \"content\": question}\n  ]\n)\nres = response.choices[0].message.content\nprint(res)\n```\n\n    Certainly! Below is a Python program that generates synthetic housing data according to your specifications. We will create a pandas DataFrame with the defined fields and characteristics.\n    \n    ```python\n    import pandas as pd\n    import random\n    \n    def generate_housing_data(num_rows):\n        data = []\n        \n        locations = [\n            ('City Center', 10000, 150),  # (location name, base price per m\u00b2, base size)\n            ('Suburban Area', 8000, 100),\n            ('Country Side', 5000, 80),\n            ('Coastal Region', 12000, 110),\n            ('Urban Neighborhood', 9000, 130)\n        ]\n        \n        for i in range(1, num_rows + 1):\n            # Randomly pick a location\n            location, base_price_per_m2, base_size = random.choice(locations)\n            \n            # Generate number of bedrooms (1 to 5)\n            number_of_bedrooms = random.randint(1, 5)\n            \n            # Calculate house size based on the number of bedrooms\n            house_size = base_size + (10 * number_of_bedrooms) + random.randint(-5, 15)  # Adding some noise\n            \n            # Calculate house price based on house size and location\n            house_price = base_price_per_m2 * house_size + random.randint(-5000, 10000)  # Adding some noise\n    \n            # Append the generated data to the list\n            data.append({\n                'id': i,\n                'house_size_m2': house_size,\n                'house_price': house_price,\n                'location': location,\n                'number_of_bedrooms': number_of_bedrooms\n            })\n    \n        # Create a pandas DataFrame\n        df = pd.DataFrame(data)\n        return df\n    \n    # Generate 100 rows of housing data\n    housing_data_df = generate_housing_data(100)\n    \n    # Show the result\n    print(housing_data_df)\n    ```\n    \n    ### Explanation:\n    - The `generate_housing_data` function creates synthetic housing data for a specified number of rows (`num_rows`).\n    - We define different locations with corresponding base prices per square meter and average house sizes.\n    - For each house, we randomly select a location, number of bedrooms, and calculate house size and price to ensure a sensible correlation between the values.\n    - Finally, we create a pandas DataFrame from the generated data and return it.\n    \n    You can run this program in your Python environment, and it will output a DataFrame containing 100 rows of synthetic housing data."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9e4b3853-1cdb-495d-99e8-85d626030bde
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9e4b3853-1cdb-495d-99e8-85d626030bde\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e26ce35e-be5f-4c81-864f-c20b71ac5f71\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f0a120eb82303e230e9335b1a2f2c04a99eb3665ade9a4402d3c92d375628e59\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4a26182-76ba-4fbc-85e3-d49b0bd57869\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"16ea581358e7938df7828f6d4365438374b63286c05a0564675d4dd9151b90b0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7673, \"end_char_idx\": 10470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "We need to make sure to parse the output of this appropriately as often there may be surrounding text to the python code. We can also explicitly ask it to state all assumptions it made about the data it's generating, however in this circumstance it told us that automatically.\n\n### 3. Multitable CSV with a python program\nFor more complex relationships however we need to make sure to specify a few more characteristics. \n\nTo create multiple different datasets which relate to each other (for example housing, location, house type), as before we would need to specify the format, schema and useful information. However, the useful information required to get good performance is higher now. It's case-specific but a good amount of things to describe would be how the datasets relate to each other, addressing the size of the datasets in relation to one another, making sure foreign and primary keys are made appropriately and ideally using previously generated datasets to populate new ones so the actual data values match where necessary.\n\n\n```python\nquestion = \"\"\"\nCreate a Python program to generate 3 different pandas dataframes.\n\n1. Housing data\nI want 100 rows. Each row should include the following fields:\n - id (incrementing integer starting at 1)\n - house size (m^2)\n - house price\n - location\n - number of bedrooms\n - house type\n + any relevant foreign keys\n\n2. Location\nEach row should include the following fields:\n - id (incrementing integer starting at 1)\n - country\n - city\n - population\n - area (m^2)\n + any relevant foreign keys\n\n 3. House types\n - id (incrementing integer starting at 1)\n - house type\n - average house type price\n - number of houses\n + any relevant foreign keys\n\nMake sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).\nMake sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.\nMake sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.\nYou can use the previously generated dataframe to generate the next dataframe.\n\"\"\"\n\nresponse = client.chat.completions.create(\n  model=datagen_model,\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n    {\"role\": \"user\", \"content\": question}\n  ]\n)\nres = response.choices[0].message.content\nprint(res)\n```\n\n    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c4a26182-76ba-4fbc-85e3-d49b0bd57869
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c4a26182-76ba-4fbc-85e3-d49b0bd57869\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e4b3853-1cdb-495d-99e8-85d626030bde\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1c7026332cd9b58ad1c5082ec71e7aebee03f7965f95ced5517bb41dfee624f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da82c1cc-0d6b-4efe-bd0b-7a55417f4fca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c85a96c211ebb27c0096724b202f1b0077c6a0a49b17f0faf10be94b59e32abe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8715, \"end_char_idx\": 13481, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "```python\nquestion = \"\"\"\nCreate a Python program to generate 3 different pandas dataframes.\n\n1. Housing data\nI want 100 rows. Each row should include the following fields:\n - id (incrementing integer starting at 1)\n - house size (m^2)\n - house price\n - location\n - number of bedrooms\n - house type\n + any relevant foreign keys\n\n2. Location\nEach row should include the following fields:\n - id (incrementing integer starting at 1)\n - country\n - city\n - population\n - area (m^2)\n + any relevant foreign keys\n\n 3. House types\n - id (incrementing integer starting at 1)\n - house type\n - average house type price\n - number of houses\n + any relevant foreign keys\n\nMake sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).\nMake sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.\nMake sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.\nYou can use the previously generated dataframe to generate the next dataframe.\n\"\"\"\n\nresponse = client.chat.completions.create(\n  model=datagen_model,\n  messages=[\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n    {\"role\": \"user\", \"content\": question}\n  ]\n)\nres = response.choices[0].message.content\nprint(res)\n```\n\n    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them.\n    \n    ```python\n    import pandas as pd\n    import numpy as np\n    \n    # Set random seed for reproducibility\n    np.random.seed(0)\n    \n    # Function to generate location DataFrame\n    def generate_location_data(num_locations):\n        locations = {\n            \"id\": range(1, num_locations + 1),\n            \"country\": np.random.choice(['USA', 'Canada', 'UK'], num_locations),\n            \"city\": np.random.choice(['New York', 'Toronto', 'London', 'Vancouver', 'Manchester'], num_locations),\n            \"population\": np.random.randint(50000, 1000000, num_locations),\n            \"area\": np.random.randint(10000, 500000, num_locations)\n        }\n        return pd.DataFrame(locations)\n    \n    # Function to generate house types DataFrame\n    def generate_house_type_data(num_house_types):\n        house_types = {\n            \"id\": range(1, num_house_types + 1),\n            \"house_type\": np.random.choice(['Detached', 'Semi-Detached', 'Terraced', 'Flat'], num_house_types),\n            \"average_house_type_price\": np.random.randint(100000, 1000000, num_house_types),\n            \"number_of_houses\": np.random.randint(10, 1000, num_house_types)\n        }\n        return pd.DataFrame(house_types)\n    \n    # Function to generate housing data DataFrame\n    def generate_housing_data(num_houses, location_df, house_type_df):\n        house_sizes = np.random.randint(50, 300, num_houses)  # size in m^2\n        location_ids = np.random.choice(location_df['id'], num_houses)\n        house_type_ids = np.random.choice(house_type_df['id'], num_houses)\n        \n        # Generate prices based on size, location, and house type\n        house_prices = (house_sizes * np.random.randint(2000, 5000, num_houses) // 10) + \\\n                       (location_ids * 1000) + \\\n                       (house_type_df.loc[house_type_ids - 1, 'average_house_type_price'].values // 4)\n        \n        housing_data = {\n            \"id\": range(1, num_houses + 1),\n            \"house_size\": house_sizes,\n            \"house_price\": house_prices,\n            \"location_id\": location_ids,\n            \"bedrooms\": np.random.randint(1, 6, num_houses),\n            \"house_type_id\": house_type_ids\n        }\n        \n        return pd.DataFrame(housing_data)\n    \n    # Generate DataFrames\n    num_locations = 10\n    num_house_types = 4\n    num_houses = 100\n    \n    location_df = generate_location_data(num_locations)\n    house_type_df = generate_house_type_data(num_house_types)\n    housing_df = generate_housing_data(num_houses, location_df, house_type_df)\n    \n    # Display the generated DataFrames\n    print(\"Location DataFrame:\")\n    print(location_df.head(), \"\\n\")\n    \n    print(\"House Types DataFrame:\")\n    print(house_type_df.head(), \"\\n\")\n    \n    print(\"Housing DataFrame:\")\n    print(housing_df.head(), \"\\n\")\n    \n    # Printing the DataFrame shapes\n    print(f\"Shapes: \\nLocation: {location_df.shape}, House Types: {house_type_df.shape}, Housing: {housing_df.shape}\")\n    ```\n    \n    ### Explanation of the Code:\n    1."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: da82c1cc-0d6b-4efe-bd0b-7a55417f4fca
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"da82c1cc-0d6b-4efe-bd0b-7a55417f4fca\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4a26182-76ba-4fbc-85e3-d49b0bd57869\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2edf50dcc5d1d1108d00fd221879b17f02ee59fa73376a29682db268cf7f9fd1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"972d5409-cc6d-4212-8a8b-d44d566828b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f3d0f24dda79e5e2b243302f6bffa8e321ed8eaa8a8816f7a15c475f6e7d2e43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13482, \"end_char_idx\": 15207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "**Location DataFrame:** \n       - Generates random locations with attributes such as country, city, population, and area.\n      \n    2. **House Types DataFrame:** \n       - Generates different types of houses along with average prices and quantity available.\n      \n    3. **Housing DataFrame:** \n       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.\n    \n    ### Output:\n    The three DataFrames generated will logically relate to one another with consistent data types and primary\u2013foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.\n\n\n### 4. Simply creating textual data\nHere we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.\n\nThe example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. \n\nSpecifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 972d5409-cc6d-4212-8a8b-d44d566828b9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"972d5409-cc6d-4212-8a8b-d44d566828b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da82c1cc-0d6b-4efe-bd0b-7a55417f4fca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1c3d80a2ccf971177b75ccc4402e4e93937b033ed70d058bdcff4d3d1c2e3a02\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e94d1f21-c796-4e00-a4ab-a8792f0fd210\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"107dcd1b16bd56ac232db0ef144fbcbcd62ee166a03bdbe078efb573ffa19887\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13482, \"end_char_idx\": 17323, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "**Location DataFrame:** \n       - Generates random locations with attributes such as country, city, population, and area.\n      \n    2. **House Types DataFrame:** \n       - Generates different types of houses along with average prices and quantity available.\n      \n    3. **Housing DataFrame:** \n       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.\n    \n    ### Output:\n    The three DataFrames generated will logically relate to one another with consistent data types and primary\u2013foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.\n\n\n### 4. Simply creating textual data\nHere we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.\n\nThe example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. \n\nSpecifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break.\n\n\n```python\noutput_string = \"\"\nfor i in range(3):\n  question = f\"\"\"\n  I am creating input output training pairs to fine tune my gpt model. The usecase is a retailer generating a description for a product from a product catalogue. I want the input to be product name and category (to which the product belongs to) and output to be description.\n  The format should be of the form:\n  1.\n  Input: product_name, category\n  Output: description\n  2.\n  Input: product_name, category\n  Output: description\n\n  Do not add any extra characters around that formatting as it will make the output parsing break.\n  Create as many training pairs as possible.\n  \"\"\"\n\n  response = client.chat.completions.create(\n    model=datagen_model,\n    messages=[\n      {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n      {\"role\": \"user\", \"content\": question}\n    ]\n  )\n  res = response.choices[0].message.content\n  output_string += res + \"\\n\" + \"\\n\"\nprint(output_string[:1000]) #displaying truncated response\n\n```\n\n    1.\n    Input: Wireless Bluetooth Headphones, Electronics\n    Output: Immerse yourself in high-quality sound with these Wireless Bluetooth Headphones, featuring active noise cancellation and a comfortable over-ear design for extended listening sessions.\n    \n    2.\n    Input: Organic Green Tea, Beverages\n    Output: Enjoy a refreshing cup of Organic Green Tea, sourced from the finest leaves, packed with antioxidants, and perfect for a healthy, invigorating boost anytime.\n    \n    3.\n    Input: Stainless Steel Kitchen Knife, Kitchenware\n    Output: Cut with precision and ease using this Stainless Steel Kitchen Knife, designed with an ergonomic handle and a sharp blade for all your culinary tasks.\n    \n    4.\n    Input: Hiking Backpack, Outdoor Gear\n    Output: Explore the great outdoors with this durable Hiking Backpack, featuring multiple compartments for optimal organization and a breathable design for ultimate comfort on long treks.\n    \n    5.\n    Input: Air Fryer, Kitchen Appliances\n    Output: Cook your favorite meals with less oil using this Air Fryer"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e94d1f21-c796-4e00-a4ab-a8792f0fd210
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e94d1f21-c796-4e00-a4ab-a8792f0fd210\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"972d5409-cc6d-4212-8a8b-d44d566828b9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c0ad067b3e86db1111a51a4977c3dd87c6c7a28c3f43676415aeb665bde52429\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"52689d2a-c4b8-439a-9ff6-7905d23b8e4c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb028639b0decedb89bbdee3201b4e6adcdf2a7455c60e40e2ed1b2b73a04829\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17326, \"end_char_idx\": 20387, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Note: the above output is truncated. And now we can parse it as below to get a list of products, categories and their descriptions. For example, let's take a look at the products it's generated.\n\n\n```python\n#regex to parse data\npattern = re.compile(r'Input:\\s*(.+?),\\s*(.+?)\\nOutput:\\s*(.+?)(?=\\n\\n|\\Z)', re.DOTALL)\nmatches = pattern.findall(output_string)\nproducts = []\ncategories = []\ndescriptions = []\n\nfor match in matches:\n    product, category, description = match\n    products.append(product.strip())\n    categories.append(category.strip())\n    descriptions.append(description.strip())\nproducts\n```\n\n\n\n\n    ['Wireless Bluetooth Headphones',\n     'Organic Green Tea',\n     'Stainless Steel Kitchen Knife',\n     'Hiking Backpack',\n     'Air Fryer',\n     \"Kids' Educational Tablet\",\n     'Bluetooth Speaker',\n     'Yoga Mat',\n     'Memory Foam Mattress',\n     'Smartwatch',\n     'Leather Wallet',\n     'Portable Phone Charger',\n     'Non-Stick Cookware Set',\n     'Pet Dog Bed',\n     'Fitness Tracker',\n     'Wireless Earbuds',\n     'Organic Green Tea',\n     'Reusable Water Bottle',\n     'Yoga Mat',\n     'Leather Wallet',\n     'Air Fryer',\n     'Gaming Mouse',\n     'Crochet Kit',\n     'Hiking Boots',\n     'Scented Candles',\n     'Bluetooth Speaker',\n     'Stainless Steel Cookware Set',\n     'Fitness Tracker',\n     'Decorative Throw Pillows',\n     'Eco-Friendly Cleaning Supplies',\n     'Wireless Noise Cancelling Headphones',\n     'Organic Green Tea',\n     'Adjustable Yoga Mat',\n     'Bluetooth Smart Scale',\n     'Stainless Steel Water Bottle',\n     'Soft Cotton Bedding Set',\n     'Multi-Functional Kitchen Blender',\n     'Eco-Friendly Reusable Bags',\n     'Portable Phone Charger',\n     'Classic Leather Wallet',\n     'Suede Chelsea Boots',\n     'Non-Stick Cookware Set',\n     'Pet-Friendly Indoor Plants',\n     'High-Protein Snack Bars',\n     'LED Desk Lamp with USB Port']\n\n\n\n\n### 5. Dealing with imbalanced or non-diverse textual data\nSome of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).\n\n\nTo increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. \n\nWe can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.\n\nFor demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 52689d2a-c4b8-439a-9ff6-7905d23b8e4c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"52689d2a-c4b8-439a-9ff6-7905d23b8e4c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e94d1f21-c796-4e00-a4ab-a8792f0fd210\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ba353dc294267170046129ee6184fe381d66d1355e9379280da9fb5564241a2f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bb8101d3-6bb5-4f7e-9467-75b699c088aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9f643c5e3db59aa48aa3ce1771d1fbf72f2826afcfb6188005c1da8be7558526\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19219, \"end_char_idx\": 22835, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "### 5. Dealing with imbalanced or non-diverse textual data\nSome of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).\n\n\nTo increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. \n\nWe can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.\n\nFor demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas.\n\n\n```python\noutput_string = \"\"\nfor i in range(3):\n  question = f\"\"\"\n  I am creating input output training pairs to fine tune my gpt model. I want the input to be product name and category and output to be description. the category should be things like: mobile phones, shoes, headphones, laptop, electronic toothbrush, etc. and also more importantly the categories should come under 4 main topics: vehicle, clothing, toiletries, food)\n  After the number of each example also state the topic area. The format should be of the form:\n  1. topic_area\n  Input: product_name, category\n  Output: description\n\n  Do not add any extra characters around that formatting as it will make the output parsing break.\n\n  Here are some helpful examples so you get the style of output correct.\n\n  1) clothing\n  Input: \"Shoe Name, Shoes\"\n  Output: \"Experience unparalleled comfort. These shoes feature a blend of modern style and the traditional superior cushioning, perfect for those always on the move.\"\n  \"\"\"\n\n  response = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[\n      {\"role\": \"system\", \"content\": \"You are a helpful assistant designed to generate synthetic data.\"},\n      {\"role\": \"user\", \"content\": question}\n    ]\n  )\n  res = response.choices[0].message.content\n  output_string += res + \"\\n\" + \"\\n\"\nprint(output_string[:1000]) #displaying truncated response\n```\n\n    1. vehicle  \n    Input: \"Tesla Model 3, Electric Car\"  \n    Output: \"The Tesla Model 3 is a revolutionary electric car with impressive range and cutting-edge technology, designed to provide an exhilarating driving experience while minimizing environmental impact.\"\n    \n    2. clothing  \n    Input: \"Nike Air Max, Shoes\"  \n    Output: \"Elevate your sneaker game with Nike Air Max. Combining iconic style with superior comfort and support, these shoes are perfect for both workouts and casual outings.\"\n    \n    3. toiletries  \n    Input: \"Oral-B Pro 1000, Electronic Toothbrush\"  \n    Output: \"Achieve a superior clean with the Oral-B Pro 1000. This electronic toothbrush features 3D cleaning action that pulsates and oscillates to remove more plaque than a regular manual toothbrush.\"\n    \n    4. food  \n    Input: \"Chobani Greek Yogurt, Yogurt\"  \n    Output: \"Indulge in a nutritious snack with Chobani Greek Yogurt. Packed with protein and delicious flavors, it\u2019s the perfect choice for a healthy breakfast or a satisfying treat anytime.\"\n    \n    5. vehicle"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bb8101d3-6bb5-4f7e-9467-75b699c088aa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bb8101d3-6bb5-4f7e-9467-75b699c088aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"52689d2a-c4b8-439a-9ff6-7905d23b8e4c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"89ba8f3ef7b297d2c9109e3f87087a07bbf2e0753e28d93d2ae5cbaf01315863\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"812a6b7e-e434-4724-bf78-021fc80f47f7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ee7ce49eda7a801dcd0d4ceb6c7d054839655873f96cd66edbe77632f040cba8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22845, \"end_char_idx\": 25758, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Note: The above output is truncated. In the example above, we would explicitly include the topic area as part of the response per example as it helps condition the proceeding output and tends to give better performance. We can also give it an actual example of what the output should look like so it gets the right idea of style of output but also to help enforce structure.\n\n\n```python\npattern = re.compile(r'(\\d+)\\.\\s*(\\w+)\\s*Input:\\s*\"(.+?),\\s*(.+?)\"\\s*Output:\\s*\"(.*?)\"', re.DOTALL)\nmatches = pattern.findall(output_string)\n\ntopics = []\nproducts = []\ncategories = []\ndescriptions = []\n\nfor match in matches:\n    number, topic, product, category, description = match\n    topics.append(topic)\n    products.append(product)\n    categories.append(category)\n    descriptions.append(description)\n\n```\n\n\n```python\nproducts\n```\n\n\n\n\n    ['Tesla Model 3',\n     'Nike Air Max',\n     'Oral-B Pro 1000',\n     'Chobani Greek Yogurt',\n     'Ford F-150',\n     \"Levi's 511\",\n     'Philips Sonicare',\n     'Quaker Oatmeal',\n     'Toyota Camry',\n     'Adidas Ultraboost',\n     'Toyota Camry',\n     'Nike Air Max',\n     'Colgate Electric Toothbrush',\n     'Blue Diamond Almonds',\n     'Harley Davidson Fat Boy',\n     'Adidas UltraBoost',\n     \"Dove Men's Body Wash\",\n     'Quaker Oats',\n     'Ford F-150',\n     \"Levi's 501 Jeans\",\n     'Tesla Model 3',\n     'Nike Air Max',\n     'Oral-B Pro 1000',\n     'Organic Almond Butter',\n     'Yamaha YZF-R3',\n     'Adidas Ultraboost',\n     'Philips Sonicare',\n     'Organic Quinoa']\n\n\n\nWe will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.\n\nWe know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.\n\nIn the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).\n\nFirst let's store our data into a pandas dataframe for ease of analysis\n\n\n\n\n```python\ndata = {\n    'Product': products,\n    'Category': categories,\n    'Description': descriptions\n}\n\ndf = pd.DataFrame(data)\n```\n\nNext let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 812a6b7e-e434-4724-bf78-021fc80f47f7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"812a6b7e-e434-4724-bf78-021fc80f47f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bb8101d3-6bb5-4f7e-9467-75b699c088aa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"85ebcc0ee837570505773a92f0261ee9ce777aae84b8c566365229448b9a8cbe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"49315982-5eb9-40eb-8db8-1b1ea6bab948\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f36525c142765e26c98342b4f3d0733fbfca7f344f4e5a7a9430c42f0f7a03d6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24355, \"end_char_idx\": 27149, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "We will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.\n\nWe know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.\n\nIn the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).\n\nFirst let's store our data into a pandas dataframe for ease of analysis\n\n\n\n\n```python\ndata = {\n    'Product': products,\n    'Category': categories,\n    'Description': descriptions\n}\n\ndf = pd.DataFrame(data)\n```\n\nNext let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar.\n\n\n```python\ndef get_embedding(text, model=\"text-embedding-3-small\"):\n    text = text.replace(\"\\n\", \" \")\n\n    response = client.embeddings.create(input=[text], model=model)\n\n    return response.data[0].embedding\n\nembedding_model = \"text-embedding-3-small\"\ndf[\"embedding\"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))\n\n# Ensure there are embeddings to concatenate\nif len(df.embedding.values) > 0:\n    matrix = np.vstack(df.embedding.values)\nelse:\n    matrix = np.array([])  # Handle the case where there are no embeddings\n\n```\n\n\n```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Product</th>\n      <th>Category</th>\n      <th>Description</th>\n      <th>embedding</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Tesla Model 3</td>\n      <td>Electric Car</td>\n      <td>The Tesla Model 3 is a revolutionary electric ...</td>\n      <td>[0.003255360759794712, -0.039260633289813995, ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Elevate your sneaker game with Nike Air Max."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 49315982-5eb9-40eb-8db8-1b1ea6bab948
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"49315982-5eb9-40eb-8db8-1b1ea6bab948\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"812a6b7e-e434-4724-bf78-021fc80f47f7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c5d4f529248801dd18d07ff88c00bb737223859be44d55c9b7119cef1f902b10\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6429b49b-d791-4506-80a6-440a6cb5d114\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e617f7016719dc7cbcd888bfc804faf08fc64faedc223d08d90222ce9daa5cd0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25761, \"end_char_idx\": 28036, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "```python\ndef get_embedding(text, model=\"text-embedding-3-small\"):\n    text = text.replace(\"\\n\", \" \")\n\n    response = client.embeddings.create(input=[text], model=model)\n\n    return response.data[0].embedding\n\nembedding_model = \"text-embedding-3-small\"\ndf[\"embedding\"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))\n\n# Ensure there are embeddings to concatenate\nif len(df.embedding.values) > 0:\n    matrix = np.vstack(df.embedding.values)\nelse:\n    matrix = np.array([])  # Handle the case where there are no embeddings\n\n```\n\n\n```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Product</th>\n      <th>Category</th>\n      <th>Description</th>\n      <th>embedding</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Tesla Model 3</td>\n      <td>Electric Car</td>\n      <td>The Tesla Model 3 is a revolutionary electric ...</td>\n      <td>[0.003255360759794712, -0.039260633289813995, ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Elevate your sneaker game with Nike Air Max. C...</td>\n      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Oral-B Pro 1000</td>\n      <td>Electronic Toothbrush</td>\n      <td>Achieve a superior clean with the Oral-B Pro 1...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chobani Greek Yogurt</td>\n      <td>Yogurt</td>\n      <td>Indulge in a nutritious snack with Chobani Gre...</td>\n      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ford F-150</td>\n      <td>Pickup Truck</td>\n      <td>The Ford F-150 is the ultimate pickup truck, d...</td>\n      <td>[0.007467855699360371, -0.05288049206137657, -...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Levi's 511</td>\n      <td>Jeans</td>\n      <td>Step out in style with Levi's 511 jeans."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6429b49b-d791-4506-80a6-440a6cb5d114
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6429b49b-d791-4506-80a6-440a6cb5d114\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"49315982-5eb9-40eb-8db8-1b1ea6bab948\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ef822fdc2c9312685937245c2e3277d173c47e8ea813c8058a04137d3e4d4c09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cab60bd-c4bd-42c6-886c-9ec73524eefa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c5bd4840b32918054101fb5947705345a4143cd4a4b884578091fc676a7af83c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27150, \"end_char_idx\": 28498, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "C...</td>\n      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Oral-B Pro 1000</td>\n      <td>Electronic Toothbrush</td>\n      <td>Achieve a superior clean with the Oral-B Pro 1...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chobani Greek Yogurt</td>\n      <td>Yogurt</td>\n      <td>Indulge in a nutritious snack with Chobani Gre...</td>\n      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ford F-150</td>\n      <td>Pickup Truck</td>\n      <td>The Ford F-150 is the ultimate pickup truck, d...</td>\n      <td>[0.007467855699360371, -0.05288049206137657, -...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Levi's 511</td>\n      <td>Jeans</td>\n      <td>Step out in style with Levi's 511 jeans. Featu...</td>\n      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Philips Sonicare</td>\n      <td>Electric Toothbrush</td>\n      <td>Discover a new level of oral care with the Phi...</td>\n      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Quaker Oatmeal</td>\n      <td>Breakfast Cereal</td>\n      <td>Start your day right with Quaker Oatmeal."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6cab60bd-c4bd-42c6-886c-9ec73524eefa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6cab60bd-c4bd-42c6-886c-9ec73524eefa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6429b49b-d791-4506-80a6-440a6cb5d114\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d956d8dab1eb90f2228c51e7ac20b9d9c8e9141153809cf20b964d250a5ee03d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2fce0d66-d71a-45d0-afb0-8ed05824735b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b0794f7573912c1050eecc2938c42d5d4eb5e05ed924fe59629c008d2b697f6f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28037, \"end_char_idx\": 29840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Featu...</td>\n      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Philips Sonicare</td>\n      <td>Electric Toothbrush</td>\n      <td>Discover a new level of oral care with the Phi...</td>\n      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Quaker Oatmeal</td>\n      <td>Breakfast Cereal</td>\n      <td>Start your day right with Quaker Oatmeal. This...</td>\n      <td>[-0.006529285106807947, 0.007865572348237038, ...</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Toyota Camry</td>\n      <td>Sedan</td>\n      <td>The Toyota Camry stands out in the sedan categ...</td>\n      <td>[-0.02088991366326809, -0.006191295105963945, ...</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Adidas Ultraboost</td>\n      <td>Running Shoes</td>\n      <td>Run like never before in the Adidas Ultraboost...</td>\n      <td>[0.02679188922047615, 0.014639599248766899, 8....</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>Toyota Camry</td>\n      <td>Car</td>\n      <td>The Toyota Camry is a reliable midsize sedan k...</td>\n      <td>[0.008056452497839928, -0.007912316359579563, ...</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Step up your sneaker game with the Nike Air Ma...</td>\n      <td>[0.03943241760134697, 0.02208484522998333, -0....</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>Colgate Electric Toothbrush</td>\n      <td>Electronic Toothbrush</td>\n      <td>Transform your oral hygiene routine with the C...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>Blue Diamond Almonds</td>\n      <td>Nuts</td>\n      <td>Snack healthy with Blue Diamond Almonds."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2fce0d66-d71a-45d0-afb0-8ed05824735b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2fce0d66-d71a-45d0-afb0-8ed05824735b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6cab60bd-c4bd-42c6-886c-9ec73524eefa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"668a01e1d28b2a7257f96188c448782c1d099bdc26b5bdf5efbddba90e9f0597\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"affff5c6-4a1f-49d4-953b-67dd05ec25e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef8dd0a14988b2c3378f6cbd7481810d95c39add1aa7c595908eac9b08eaf7a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29841, \"end_char_idx\": 31613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "These...</td>\n      <td>[-0.013289917260408401, 0.036334190517663956, ...</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>Harley Davidson Fat Boy</td>\n      <td>Motorcycle</td>\n      <td>Experience the thrill of the open road with th...</td>\n      <td>[0.012365399859845638, 0.03552943095564842, -0...</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>Adidas UltraBoost</td>\n      <td>Sneakers</td>\n      <td>Enjoy a perfect blend of comfort and performan...</td>\n      <td>[0.013107392005622387, 0.02963760495185852, -0...</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>Dove Men's Body Wash</td>\n      <td>Body Wash</td>\n      <td>Refresh and hydrate your skin with Dove Men's ...</td>\n      <td>[0.03760576993227005, -0.008475445210933685, -...</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>Quaker Oats</td>\n      <td>Oats</td>\n      <td>Start your day right with Quaker Oats. Packed ...</td>\n      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>Ford F-150</td>\n      <td>Truck</td>\n      <td>The Ford F-150 is a durable and dependable tru...</td>\n      <td>[0.023461222648620605, -0.026651185005903244, ...</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>Levi's 501 Jeans</td>\n      <td>Jeans</td>\n      <td>Discover the timeless style of Levi's 501 Jean...</td>\n      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>Tesla Model 3</td>\n      <td>Mobile Phones</td>\n      <td>Explore the future of driving with the Tesla M...</td>\n      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Step up your game with the Nike Air Max."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: affff5c6-4a1f-49d4-953b-67dd05ec25e0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"affff5c6-4a1f-49d4-953b-67dd05ec25e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2fce0d66-d71a-45d0-afb0-8ed05824735b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"0c4ef7676d35ad51d9ff9881c2d0825c195fd8915df7656ce6fd732ed22c3da6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"42a1def6-820a-4294-ad5e-9b733a261a0f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a775869be6e9f7ffad94df167ce3a3f1d7bec72f2488b49f32026a44645a77b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30737, \"end_char_idx\": 33087, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Packed ...</td>\n      <td>[-0.00903365109115839, 0.00896345917135477, 0....</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>Ford F-150</td>\n      <td>Truck</td>\n      <td>The Ford F-150 is a durable and dependable tru...</td>\n      <td>[0.023461222648620605, -0.026651185005903244, ...</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>Levi's 501 Jeans</td>\n      <td>Jeans</td>\n      <td>Discover the timeless style of Levi's 501 Jean...</td>\n      <td>[0.003762696636840701, 0.02275814116001129, -0...</td>\n    </tr>\n    <tr>\n      <th>20</th>\n      <td>Tesla Model 3</td>\n      <td>Mobile Phones</td>\n      <td>Explore the future of driving with the Tesla M...</td>\n      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>\n    </tr>\n    <tr>\n      <th>21</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Step up your game with the Nike Air Max. Desig...</td>\n      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>\n    </tr>\n    <tr>\n      <th>22</th>\n      <td>Oral-B Pro 1000</td>\n      <td>Electronic Toothbrush</td>\n      <td>Achieve a superior clean with the Oral-B Pro 1...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n    </tr>\n    <tr>\n      <th>23</th>\n      <td>Organic Almond Butter</td>\n      <td>Food</td>\n      <td>Indulge in the creamy goodness of Organic Almo...</td>\n      <td>[-0.014613640494644642, -0.002179765608161688,...</td>\n    </tr>\n    <tr>\n      <th>24</th>\n      <td>Yamaha YZF-R3</td>\n      <td>Mobile Phones</td>\n      <td>Introducing the Yamaha YZF-R3, the ultimate sp...</td>\n      <td>[0.03703858703374863, 0.03407958149909973, 0.0...</td>\n    </tr>\n    <tr>\n      <th>25</th>\n      <td>Adidas Ultraboost</td>\n      <td>Shoes</td>\n      <td>Discover the Adidas Ultraboost, a shoe that of...</td>\n      <td>[0.03944042697548866, 0.022062409669160843, -0...</td>\n    </tr>\n    <tr>\n      <th>26</th>\n      <td>Philips Sonicare</td>\n      <td>Electronic Toothbrush</td>\n      <td>Experience the dental care revolution with Phi...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n    </tr>\n    <tr>\n      <th>27</th>\n      <td>Organic Quinoa</td>\n      <td>Food</td>\n      <td>Nourish your body with Organic Quinoa, a nutri...</td>\n      <td>[-0.014613640494644642, -0.002179765608161688,...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 42a1def6-820a-4294-ad5e-9b733a261a0f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"42a1def6-820a-4294-ad5e-9b733a261a0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"affff5c6-4a1f-49d4-953b-67dd05ec25e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"aae4ded9088f0dfe449517050ed569c7ca743825f9a35924eed5080eb1a89db5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"768179ab-7d6e-445b-9e0e-03480b7ebde5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"519a5446b5d6e76792329059674cb699b4dcea207584a004c9e5bb695cb12a2f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33091, \"end_char_idx\": 35530, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Now we perform the elbow method. \n\n\n```python\n# Determine the optimal number of clusters using the elbow method\ninertias = []\nrange_of_clusters = range(1, 13)  # Adjust the range as necessary\n\nfor n_clusters in range_of_clusters:\n    kmeans = KMeans(n_clusters=n_clusters, init=\"k-means++\", random_state=42, n_init=10)\n    kmeans.fit(matrix)\n    inertias.append(kmeans.inertia_)\n\n```\n\nThis will output a chart for us in which we have to visually tell where the optimal cluster point is. We can see below that we see a gradual decrease of inertia rather than a sharp elbow but the point of steepest decrease appears to occur around 3, 4 or 5 clusters which lines up with our expectations given our prompt. \n\n\n```python\n# Plotting the elbow plot\nplt.figure(figsize=(10, 6))\nplt.plot(range_of_clusters, inertias, '-o')\nplt.title('Elbow Method to Determine Optimal Number of Clusters')\nplt.xlabel('Number of Clusters')\nplt.ylabel('Inertia')\nplt.xticks(range_of_clusters)\nplt.show()\n```\n\n\n    \n![png](output_32_0.png)\n    \n\n\n![elbow_chart](../images/elbow_chart.png)\n\nFor demonstration purposes we will pick 5 as the optimal cluster number to show it doesn't matter exactly where we pick it as long as we are approximately right. There are numerous correct ways to categorize data. We also store which cluster each data point belongs to.\n\n\n```python\nn_clusters = 5\n\nkmeans = KMeans(n_clusters=n_clusters, init=\"k-means++\", random_state=42)\nkmeans.fit(matrix)\nlabels = kmeans.labels_\ndf[\"Cluster\"] = labels\n```\n\nWe will analyze the cluster data now. There are two separate things we will look to address. 1. imbalanced data, 2. Expanding the data distribution.\n\nFirst for imbalanced data we count the number of examples in each cluster. Then we select a few examples from each cluster at random and ask the LLM what topics these map to. \n\n\n```python\ncluster_counts = df[\"Cluster\"].value_counts().sort_index()\nprint(cluster_counts)\n```\n\n    Cluster\n    0    5\n    1    7\n    2    8\n    3    6\n    4    2\n    Name: count, dtype: int64\n\n\nWe can see the topics found here:\nEco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel\nmatch well enough but not exactly to our initial prompt of:\nvehicle, clothing, toiletries, food.\n\nAs we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.\n\n\n```python\ndf\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 768179ab-7d6e-445b-9e0e-03480b7ebde5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"768179ab-7d6e-445b-9e0e-03480b7ebde5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"42a1def6-820a-4294-ad5e-9b733a261a0f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cc9044e103aed33854746200760770356f9de0a071074c663a7984ae56bac56a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"721c4fb9-f07a-4ee7-bf05-e79124ac51b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab588b90d6dc24973fdb2a10671cbe85c01658c06925f07939fd38007b93dadf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34080, \"end_char_idx\": 36391, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "![png](output_32_0.png)\n    \n\n\n![elbow_chart](../images/elbow_chart.png)\n\nFor demonstration purposes we will pick 5 as the optimal cluster number to show it doesn't matter exactly where we pick it as long as we are approximately right. There are numerous correct ways to categorize data. We also store which cluster each data point belongs to.\n\n\n```python\nn_clusters = 5\n\nkmeans = KMeans(n_clusters=n_clusters, init=\"k-means++\", random_state=42)\nkmeans.fit(matrix)\nlabels = kmeans.labels_\ndf[\"Cluster\"] = labels\n```\n\nWe will analyze the cluster data now. There are two separate things we will look to address. 1. imbalanced data, 2. Expanding the data distribution.\n\nFirst for imbalanced data we count the number of examples in each cluster. Then we select a few examples from each cluster at random and ask the LLM what topics these map to. \n\n\n```python\ncluster_counts = df[\"Cluster\"].value_counts().sort_index()\nprint(cluster_counts)\n```\n\n    Cluster\n    0    5\n    1    7\n    2    8\n    3    6\n    4    2\n    Name: count, dtype: int64\n\n\nWe can see the topics found here:\nEco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel\nmatch well enough but not exactly to our initial prompt of:\nvehicle, clothing, toiletries, food.\n\nAs we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.\n\n\n```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Product</th>\n      <th>Category</th>\n      <th>Description</th>\n      <th>embedding</th>\n      <th>Cluster</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Tesla Model 3</td>\n      <td>Electric Car</td>\n      <td>The Tesla Model 3 is a revolutionary electric ...</td>\n      <td>[0.003255360759794712, -0.039260633289813995, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Elevate your sneaker game with Nike Air Max."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 721c4fb9-f07a-4ee7-bf05-e79124ac51b5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"721c4fb9-f07a-4ee7-bf05-e79124ac51b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"768179ab-7d6e-445b-9e0e-03480b7ebde5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2688c9dc0c2bc7561472fe3f06bf65a10502f11b56b9627b55efc4a3f4c84bdb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f14180b8bdbd0b4c68e75ec73f6c4b0666f7dfe6bc3f34e8971b82290db22811\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 35120, \"end_char_idx\": 37346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "We can see the topics found here:\nEco-friendly Transportation, Luxury and Leisure Items, Personal Care Products, Electronic Toothbrushes and Clothing and Apparel\nmatch well enough but not exactly to our initial prompt of:\nvehicle, clothing, toiletries, food.\n\nAs we chose 5 clusters, it split up toiletries into Skincare and Personal Care which doesn't affect us too much further downstream.\n\n\n```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Product</th>\n      <th>Category</th>\n      <th>Description</th>\n      <th>embedding</th>\n      <th>Cluster</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Tesla Model 3</td>\n      <td>Electric Car</td>\n      <td>The Tesla Model 3 is a revolutionary electric ...</td>\n      <td>[0.003255360759794712, -0.039260633289813995, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Elevate your sneaker game with Nike Air Max. C...</td>\n      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Oral-B Pro 1000</td>\n      <td>Electronic Toothbrush</td>\n      <td>Achieve a superior clean with the Oral-B Pro 1...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chobani Greek Yogurt</td>\n      <td>Yogurt</td>\n      <td>Indulge in a nutritious snack with Chobani Gre...</td>\n      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ford F-150</td>\n      <td>Pickup Truck</td>\n      <td>The Ford F-150 is the ultimate pickup truck, d...</td>\n      <td>[0.007467855699360371, -0.05288049206137657, -...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Levi's 511</td>\n      <td>Jeans</td>\n      <td>Step out in style with Levi's 511 jeans."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"721c4fb9-f07a-4ee7-bf05-e79124ac51b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f42935f28f352748287e1c7611591b620a43558e297a3eb3bbd3a2e92a9d1a42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a9aecde-b131-49c8-bc9e-f8d9a642957b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f1add170365f3379c540b1f73b5ada90f796f3a47fb5d8dd05e996f140f33097\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 36392, \"end_char_idx\": 37842, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "C...</td>\n      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Oral-B Pro 1000</td>\n      <td>Electronic Toothbrush</td>\n      <td>Achieve a superior clean with the Oral-B Pro 1...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chobani Greek Yogurt</td>\n      <td>Yogurt</td>\n      <td>Indulge in a nutritious snack with Chobani Gre...</td>\n      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Ford F-150</td>\n      <td>Pickup Truck</td>\n      <td>The Ford F-150 is the ultimate pickup truck, d...</td>\n      <td>[0.007467855699360371, -0.05288049206137657, -...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Levi's 511</td>\n      <td>Jeans</td>\n      <td>Step out in style with Levi's 511 jeans. Featu...</td>\n      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Philips Sonicare</td>\n      <td>Electric Toothbrush</td>\n      <td>Discover a new level of oral care with the Phi...</td>\n      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Quaker Oatmeal</td>\n      <td>Breakfast Cereal</td>\n      <td>Start your day right with Quaker Oatmeal."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8a9aecde-b131-49c8-bc9e-f8d9a642957b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8a9aecde-b131-49c8-bc9e-f8d9a642957b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"392e7fc4-0cf3-476a-98bd-818b172251a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a3095ee8483c3693c0c86dd6e8459a36bc20a0de84f06d32bcb0db2493fe2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea6ea80c-bec5-4e25-9aba-d9e11b5f6de0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb\", \"title\": \"Synthetic Data generation (Part 1)\", \"tokens\": 15064, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e1d43104c8f95caa7d87bed0e13e52743e28644f7439a15bd84752cdc1f41985\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2dc16581-17fe-4db6-810a-d7a9f2f489e1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"99a44190d96225c11abbd319e7096947f4d857f95b2208bcb53cf232008e8975\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37347, \"end_char_idx\": 39286, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "ref_doc_id": "392e7fc4-0cf3-476a-98bd-818b172251a4", "chroma:document": "Featu...</td>\n      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Philips Sonicare</td>\n      <td>Electric Toothbrush</td>\n      <td>Discover a new level of oral care with the Phi...</td>\n      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Quaker Oatmeal</td>\n      <td>Breakfast Cereal</td>\n      <td>Start your day right with Quaker Oatmeal. This...</td>\n      <td>[-0.006529285106807947, 0.007865572348237038, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Toyota Camry</td>\n      <td>Sedan</td>\n      <td>The Toyota Camry stands out in the sedan categ...</td>\n      <td>[-0.02088991366326809, -0.006191295105963945, ...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Adidas Ultraboost</td>\n      <td>Running Shoes</td>\n      <td>Run like never before in the Adidas Ultraboost...</td>\n      <td>[0.02679188922047615, 0.014639599248766899, 8....</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>Toyota Camry</td>\n      <td>Car</td>\n      <td>The Toyota Camry is a reliable midsize sedan k...</td>\n      <td>[0.008056452497839928, -0.007912316359579563, ...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>Nike Air Max</td>\n      <td>Shoes</td>\n      <td>Step up your sneaker game with the Nike Air Ma...</td>\n      <td>[0.03943241760134697, 0.02208484522998333, -0....</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>Colgate Electric Toothbrush</td>\n      <td>Electronic Toothbrush</td>\n      <td>Transform your oral hygiene routine with the C...</td>\n      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>Blue Diamond Almonds</td>\n      <td>Nuts</td>\n      <td>Snack healthy with Blue Diamond Almonds."}


