title: python scraping max retries exceeded with url
body: <p>I am trying to scrape a website (~3000 pages) once a day. My code for the requests is:</p>
  
  <pre><code>#Return soup of page of Carsales ads
  def getsoup(pagelimit,offset):
      url = "http://www.carsales.com.au/cars/results?q=%28Service%3D%5BCarsales%5D%26%28%28%28SiloType%3D%5BDemo%20and%20near%20new%20cars%5D%7CSiloType%3D%5BDealer%20used%20cars%5D%29%7CSiloType%3D%5BDemo%20and%20near%20new%20cars%5D%29%7CSiloType%3D%5BPrivate%20seller%20cars%5D%29%29&amp;cpw=1&amp;sortby={0}&amp;limit={1}&amp;offset={2}".format('Price',str(pagelimit),str(offset))
      #Sortby options: LastUpdated,Price,
      r = requests.get(url, headers)
      soup = BeautifulSoup(r.text, "html5lib") #"html.parser"
      totalpages = int(soup.find("div",  class_="pagination").text.split(' of ',1)[1].split('\n', 1)[0])
      currentpage = int(soup.find("div",  class_="pagination").text.split('Page ',1)[1].split(' of', 1)[0])   
      return (soup, totalpages, currentpage)
  
  adscrape = []
  #Run through all search result pages, appending ads to adscrape    
  while currentpage &lt; totalpages:
      soup, totalpages, currentpage = getsoup(pagelimit,offset)
      print 'Page: {0} of {1}. Offset is {2}.'.format(currentpage,totalpages,offset)
      adscrape.extend(getpageads(soup,offset))
      offset = offset+pagelimit
      # sleep(1) 
  </code></pre>
  
  <p>I have previously run it successfully with no sleep() function to rate limit. However now I am getting errors midway through execution, it does this whether or not sleep(1) is active in the code or not:</p>
  
  <pre><code>...    
  Page: 1523 of 2956. Offset is 91320.
  Page: 1524 of 2966. Offset is 91380.
  Page: 1525 of 2956. Offset is 91440.
  Traceback (most recent call last):
    File "D:\Google Drive\pythoning\carsales\carsales_v2.py", line 82, in &lt;module&gt;
      soup, totalpages, currentpage = getsoup(pagelimit,offset)
    File "D:\Google Drive\pythoning\carsales\carsales_v2.py", line 28, in getsoup
      r = requests.get(url, headers)
    File "C:\Python27\lib\site-packages\requests\api.py", line 69, in get
      return request('get', url, params=params, **kwargs)
    File "C:\Python27\lib\site-packages\requests\api.py", line 50, in request
      response = session.request(method=method, url=url, **kwargs)
    File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
      resp = self.send(prep, **send_kwargs)
    File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
      r = adapter.send(request, **kwargs)
    File "C:\Python27\lib\site-packages\requests\adapters.py", line 423, in send
      raise ConnectionError(e, request=request)
  requests.exceptions.ConnectionError: HTTPConnectionPool(host='www.carsales.com.au', port=80): Max retries exceeded with url: /cars/results?q=%28Service%3D%5BCarsales%5D%26%28%28%28SiloType%3D%5BDemo%20and%20near%20new%20cars%5D%7CSiloType%3D%5BDealer%20used%20cars%5D%29%7CSiloType%3D%5BDemo%20and%20near%20new%20cars%5D%29%7CSiloType%3D%5BPrivate%20seller%20cars%5D%29%29&amp;cpw=1&amp;sortby=Price&amp;limit=60&amp;offset=91500&amp;user-agent=Mozilla%2F5.0 (Caused by NewConnectionError('&lt;requests.packages.urllib3.connection.HTTPConnection object at 0x000000001A097710&gt;: Failed to establish a new connection: [Errno 11004] getaddrinfo failed',))
  &lt;&lt;&lt; Process finished. (Exit code 1)
  </code></pre>
  
  <p>I'm assuming this is due to hitting the server with too many requests in a certain time. If so, how can I avoid this without making the script take hours and hours to run? What is the normal practice for scraping websites to counter this issue?</p>