title: How to avoid Max error retries exceeded in scraping with Python?
body: <p>In Python 3 I made a program to scrape the contents of table rows across multiple pages (97893). Each page has a set of tables, so I browse each page and fill a list</p>
  
  <p>The page that starts is this: <a href="http://www.portaltransparencia.gov.br/PortalComprasDiretasFavorecido.asp?TipoPesquisa=2&amp;Ano=2017&amp;Pagina=1" rel="nofollow noreferrer">http://www.portaltransparencia.gov.br/PortalComprasDiretasFavorecido.asp?TipoPesquisa=2&amp;Ano=2017&amp;Pagina=1</a></p>
  
  <pre><code>from bs4 import BeautifulSoup
  import requests
  
  def sopa(link):
      res = requests.get(link)
      soup =  BeautifulSoup(res.text, "lxml")
      table = soup.select("table")[1]
      conjunto = table.findAll("tr")
      return conjunto
  
  planilha = []
  for i in range(1,97893):
      link = "http://www.portaltransparencia.gov.br/PortalComprasDiretasFavorecido.asp?TipoPesquisa=2&amp;Ano=2017&amp;Pagina="
      link = link + str(i)
      conjunto = sopa(link)
      conta = 0
      for linha in conjunto:
          if conta &gt; 0:
              documento = linha.find("td", {"class": "firstChild"}, {"style": "white-space: nowrap;"}).text.strip()
              nome = linha.find("a").text.strip()
              valor = linha.find("td", {"class": "colunaValor"}).text.strip()
              dicionario = {"documento": documento, "nome": nome, "valor": valor}
              planilha.append(dicionario)
          conta = conta + 1
  </code></pre>
  
  <p>But on page 458 this error appeared:</p>
  
  <pre><code>ConnectionError: HTTPConnectionPool(host='www.portaltransparencia.gov.br', port=80): 
  Max retries exceeded with url: /PortalComprasDiretasFavorecido.asp?TipoPesquisa=2&amp;Ano=2017&amp;Pagina=458 
  (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7feff9ce8f98&gt;: 
  Failed to establish a new connection: [Errno -2] Name or service not known',))
  </code></pre>
  
  <p>I believe I was overthrown because I accessed many pages in sequence. Please, does anyone know how I can avoid this?</p>