Spaces:
Running
Running
File size: 4,476 Bytes
cfd3735 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
{
"cells": [
{
"cell_type": "markdown",
"id": "f6574496-b360-4ffa-9523-7fd34a590164",
"metadata": {},
"source": [
"# How to use the async API for LLMs\n",
"\n",
"LangChain provides async support for LLMs by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
"\n",
"Async support is particularly useful for calling multiple LLMs concurrently, as these calls are network-bound. Currently, `OpenAI`, `PromptLayerOpenAI`, `ChatOpenAI` and `Anthropic` are supported, but async support for other LLMs is on the roadmap.\n",
"\n",
"You can use the `agenerate` method to call an OpenAI LLM asynchronously."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5e49e96c-0f88-466d-b3d3-ea0966bdf19e",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, how about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about yourself?\n",
"\n",
"\n",
"I'm doing well, thank you! How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you! How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\u001b[1mConcurrent executed in 1.39 seconds.\u001b[0m\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about yourself?\n",
"\n",
"\n",
"I'm doing well, thanks for asking. How about you?\n",
"\n",
"\n",
"I'm doing well, thanks! How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about you?\n",
"\n",
"\n",
"I'm doing well, thank you. How about yourself?\n",
"\n",
"\n",
"I'm doing well, thanks for asking. How about you?\n",
"\u001b[1mSerial executed in 5.77 seconds.\u001b[0m\n"
]
}
],
"source": [
"import time\n",
"import asyncio\n",
"\n",
"from langchain.llms import OpenAI\n",
"\n",
"def generate_serially():\n",
" llm = OpenAI(temperature=0.9)\n",
" for _ in range(10):\n",
" resp = llm.generate([\"Hello, how are you?\"])\n",
" print(resp.generations[0][0].text)\n",
"\n",
"\n",
"async def async_generate(llm):\n",
" resp = await llm.agenerate([\"Hello, how are you?\"])\n",
" print(resp.generations[0][0].text)\n",
"\n",
"\n",
"async def generate_concurrently():\n",
" llm = OpenAI(temperature=0.9)\n",
" tasks = [async_generate(llm) for _ in range(10)]\n",
" await asyncio.gather(*tasks)\n",
"\n",
"\n",
"s = time.perf_counter()\n",
"# If running this outside of Jupyter, use asyncio.run(generate_concurrently())\n",
"await generate_concurrently() \n",
"elapsed = time.perf_counter() - s\n",
"print('\\033[1m' + f\"Concurrent executed in {elapsed:0.2f} seconds.\" + '\\033[0m')\n",
"\n",
"s = time.perf_counter()\n",
"generate_serially()\n",
"elapsed = time.perf_counter() - s\n",
"print('\\033[1m' + f\"Serial executed in {elapsed:0.2f} seconds.\" + '\\033[0m')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e1d3a966-3a27-44e8-9441-ed72f01b86f4",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|