title: asyncio web scraping 101: fetching multiple urls with aiohttp
body: <p>In earlier question, one of authors of <code>aiohttp</code> kindly suggested way to <a href="https://stackoverflow.com/q/35879769/">fetch multiple urls with aiohttp</a> using the new <code>async with</code> syntax from <code>Python 3.5</code>:</p>
  
  <pre><code>import aiohttp
  import asyncio
  
  async def fetch(session, url):
      with aiohttp.Timeout(10):
          async with session.get(url) as response:
              return await response.text()
  
  async def fetch_all(session, urls, loop):
      results = await asyncio.wait([loop.create_task(fetch(session, url))
                                    for url in urls])
      return results
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      # breaks because of the first url
      urls = ['http://SDFKHSKHGKLHSKLJHGSDFKSJH.com',
              'http://google.com',
              'http://twitter.com']
      with aiohttp.ClientSession(loop=loop) as session:
          the_results = loop.run_until_complete(
              fetch_all(session, urls, loop))
          # do something with the the_results
  </code></pre>
  
  <p>However when one of the <code>session.get(url)</code> requests breaks (as above because of <code>http://SDFKHSKHGKLHSKLJHGSDFKSJH.com</code>) the error is not handled and the whole thing breaks.</p>
  
  <p>I looked for ways to insert tests about the result of <code>session.get(url)</code>, for instance looking for places for a <code>try ... except ...</code>, or for a <code>if response.status != 200:</code> but I am just not understanding how to work with <code>async with</code>, <code>await</code> and the various objects.</p>
  
  <p>Since <code>async with</code> is still very new there are not many examples. It would be very helpful to many people if an <code>asyncio</code> wizard could show how to do this. After all one of the first things most people will want to test with <code>asyncio</code> is getting multiple resources concurrently.</p>
  
  <p><strong>Goal</strong></p>
  
  <p>The goal is that we can inspect <code>the_results</code> and quickly see either: </p>
  
  <ul>
  <li>this url failed (and why: status code, maybe exception name),  or</li>
  <li>this url worked, and here is a useful response object</li>
  </ul>