title: Multiple URL requests to API without getting error from urllib2 or requests
body: <p>I am trying to get data from different APIs. They are received in JSON format, stored in SQLite and afterwards parsed.</p>
  <p>The issue I am having is that when sending many requests I eventually receive an error, even if I am using <code>time.sleep</code> between requests.</p>
  <h1>Usual approach</h1>
  <p>My code looks like the one below, where this would be inside a loop and the url to be opened would be changing:</p>
  <pre><code>base_url = 'https://www.whateversite.com/api/index.php?'
  custom_url = 'variable_text1' + &amp; + 'variable_text2' 
  
  url = base_url + custom_urls #url will be changing
  
  time.sleep(1)
  data = urllib2.urlopen(url).read() 
  </code></pre>
  <p>This runs thousands of times inside the loop. The problem comes after the script has been running for a while (up to a couple of hours), then I get the following errors or similar ones:</p>
  <pre><code>    data = urllib2.urlopen(url).read()
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 127, in urlopen
      return _opener.open(url, data, timeout)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 404, in open
      response = self._open(req, data)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 422, in _open
      '_open', req)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 382, in _call_chain
      result = func(*args)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 1222, in https_open
      return self.do_open(httplib.HTTPSConnection, req)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py&quot;, line 1184, in do_open
      raise URLError(err)
  urllib2.URLError: &lt;urlopen error [Errno 8] nodename nor servname provided, or not known&gt;
  </code></pre>
  <p>or</p>
  <pre><code>    uh = urllib.urlopen(url)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py&quot;, line 87, in urlopen
      return opener.open(url)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py&quot;, line 208, in open
      return getattr(self, name)(url)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py&quot;, line 437, in open_https
      h.endheaders(data)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py&quot;, line 969, in endheaders
      self._send_output(message_body)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py&quot;, line 829, in _send_output
      self.send(msg)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py&quot;, line 791, in send
      self.connect()
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py&quot;, line 1172, in connect
      self.timeout, self.source_address)
    File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py&quot;, line 553, in create_connection
      for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  IOError: [Errno socket error] [Errno 8] nodename nor servname provided, or not known
  </code></pre>
  <p>I believe this happens because the modules throw an error at some point if used too often in a short period of time.</p>
  <p>From what I've read in many different threads about <a href="https://stackoverflow.com/questions/2018026/what-are-the-differences-between-the-urllib-urllib2-and-requests-module">which module is better</a>, I think for my needs all would work and the main lever to choose one is that it can open as many url's as possible. In my experience, <code>urllib</code> and <code>urllib2</code> are better than <code>requests</code> for that, as <code>requests</code> crashed in less time.</p>
  <p>Assuming that I do not want to increase the waiting time used in <code>time.sleep</code>, these are the solutions I thought so far:</p>
  <h1>Possible solutions?</h1>
  <h2>A</h2>
  <p>I thought of combining all different modules. That would be:</p>
  <ul>
  <li>Start, for instance, with <code>requests</code>.</li>
  <li>After a specific time or when error is thrown, switch automatically
  to <code>urllib2</code></li>
  <li>After a specific time or when error is thrown, switch automatically to other modules (such as <code>httplib2</code> or <code>urllib</code>) or back to <code>requests</code></li>
  <li>And so on...</li>
  </ul>
  <h2>B</h2>
  <p>Use <code>try .. except</code> block to handle that exception, as suggested <a href="https://stackoverflow.com/questions/20568216/python-handling-socket-error-errno-104-connection-reset-by-peer">here</a>.</p>
  <h2>C</h2>
  <p>I've also read about <a href="https://stackoverflow.com/questions/34512646/how-to-speed-up-api-requests">sending multiple requests at once or in parallel</a>. I don't know how that exactly works and if it could actually be useful</p>
  <hr />
  <p>However, I am not convinced about any of these solutions.</p>
  <p>Can you think of any more elegant and/or efficient solution to deal with this error?</p>
  <p><em>I'm with Python 2.7</em></p>