title: How to get httpx.gather() with return_exceptions=True to complete the Queue of tasks when the exception count exceeds the worker count?
body: <p>I'm using asyncio in concert with the httpx.AsyncClient for the first time and trying to figure out how to complete my list of tasks when some number of them may fail.  I'm using a pattern I found in a few places where I populate an asyncio Queue with coroutine functions, and have a set of workers process that queue from inside asyncio.gather.  Normally, if the function doing the work raises an exception, you'll see the whole script just fail during that processing, and report the exception along with a <code>RuntimeWarning: coroutine foo was never awaited</code>, indicating that you never finished your list.</p>
  <p>I found the <code>return_exceptions</code> option for asyncio.gather, and that has helped, but not completely.  my script will still die after I've gotten the exception the same number of times as the total number of workers that I've thrown into my call to <code>gather</code>.  The following is a simple script that demonstrates the problem.</p>
  <pre><code>from httpx import AsyncClient, Timeout
  from asyncio import run, gather, Queue as asyncio_Queue
  from random import choice
  
  
  async def process_url(client, url):
      &quot;&quot;&quot;
      opens the URL and pulls a header attribute
      randomly raises an exception to demonstrate my problem
      &quot;&quot;&quot;
      if choice([True, False]):
          await client.get(url)
          print(f'retrieved url {url}')
      else:
          raise AssertionError(f'generated error for url {url}')
  
  
  async def main(worker_count, urls):
      &quot;&quot;&quot;
      orchestrates the workers that call process_url
      &quot;&quot;&quot;
      httpx_timeout = Timeout(10.0, read=20.0)
      async with AsyncClient(timeout=httpx_timeout, follow_redirects=True) as client:
          tasks = asyncio_Queue(maxsize=0)
          for url in urls:
              await tasks.put(process_url(client, url))
  
          async def worker():
              while not tasks.empty():
                  await tasks.get_nowait()
  
          results = await gather(*[worker() for _ in range(worker_count)], return_exceptions=True)
          return results
  
  if __name__ == '__main__':
      urls = ['https://stackoverflow.com/questions',
              'https://stackoverflow.com/jobs',
              'https://stackoverflow.com/tags',
              'https://stackoverflow.com/users',
              'https://www.google.com/',
              'https://www.bing.com/',
              'https://www.yahoo.com/',
              'https://www.foxnews.com/',
              'https://www.cnn.com/',
              'https://www.npr.org/',
              'https://www.opera.com/',
              'https://www.mozilla.org/en-US/firefox/',
              'https://www.google.com/chrome/',
              'https://www.epicbrowser.com/'
              ]
      print(f'processing {len(urls)} urls')
      run_results = run(main(4, urls))
      print('\n'.join([str(rr) for rr in run_results]))
  </code></pre>
  <p>one run of this script outputs:</p>
  <pre><code>processing 14 urls
  retrieved url https://stackoverflow.com/tags
  retrieved url https://stackoverflow.com/jobs
  retrieved url https://stackoverflow.com/users
  retrieved url https://www.bing.com/
  generated error for url https://stackoverflow.com/questions
  generated error for url https://www.foxnews.com/
  generated error for url https://www.google.com/
  generated error for url https://www.yahoo.com/
  sys:1: RuntimeWarning: coroutine 'process_url' was never awaited
  
  Process finished with exit code 0
  </code></pre>
  <p>Here you see that we got through 8 of the total 14 urls, but by the time we reached 4 errors, the script wrapped up and ignored the rest of the urls.</p>
  <p>What I want to do is have the script complete the full set of urls, but inform me of the errors at the end.  Is there a way to do this here?  It may be that I'll have to wrap everything in <code>process_url()</code> inside a <code>try/except</code> block and use something like aiofile to dump them out in the end?</p>
  <p><strong>Update</strong>
  To be clear, this demo script is a simplification of what I'm really doing.  My real script is hitting a small number of server api endpoints a few hundred thousand times.  The purpose of using the set of workers is to avoid overwhelming the server I'm hitting [it's a test server, not production, so it's not intended to handle huge volumes of requests, though the number is greater than 4 8-)].  I'm open to learning about alternatives.</p>