title: python/httpx/asyncio: httpx.RemoteProtocolError: Server disconnected without sending a response
body: <p>I am attempting to optimize a simple web scraper that I made. It gets a list of urls from a table on a main page and then goes to each of those &quot;sub&quot; urls and gets information from those pages. I was able to successfully write it synchronously and using <code>concurrent.futures.ThreadPoolExecutor()</code>. However, I am trying to optimize it to use <code>asyncio</code> and <code>httpx</code> as these seem to be very fast for making hundreds of http requests.</p>
  <p>I wrote the following script using <code>asyncio</code> and <code>httpx</code> however, I keep getting the following errors:</p>
  <pre><code>httpcore.RemoteProtocolError: Server disconnected without sending a response.
  
  RuntimeError: The connection pool was closed while 4 HTTP requests/responses were still in-flight.
  </code></pre>
  <p>It appears that I keep losing connection when I run the script. I even attempted running a synchronous version of it and get the same error. I was thinking that the remote server was blocking my requests, however, <strong>I am able to run my original program and go to each of the urls from the same IP address without issue</strong>.</p>
  <p>What would cause this exception and how do you fix it?</p>
  <pre><code>import httpx
  import asyncio
  
  async def get_response(client, url):
      resp = await client.get(url, headers=random_user_agent()) # Gets a random user agent.
      html = resp.text
      return html
  
  
  async def main():
      async with httpx.AsyncClient() as client:
          tasks = []
  
          # Get list of urls to parse.
          urls = get_events('https://main-url-to-parse.com')
          
          # Get the responses for the detail page for each event
          for url in urls:
              tasks.append(asyncio.ensure_future(get_response(client, url)))
              
          detail_responses = await asyncio.gather(*tasks)
  
          for resp in detail_responses:
              event = get_details(resp) # Parse url and get desired info
          
  asyncio.run(main())
  </code></pre>