title: Google scraping using python - requests: How to avoid being blocked due to many requests?
body: <p>For a school project I need get the web addresses of 200 companies (based on a list). My script is working fine, but when I'm around the company 80, I get blocked by google. This is the message that I'm getting. </p>
  
  <pre><code>&gt; Our systems have detected unusual traffic from your computer network. 
  &gt; This page checks to see if it's really you sending the requests, and
  &gt; not a robot.  &lt;a href="#"
  &gt; onclick="document.getElementById('infoDiv').style.display='block'
  </code></pre>
  
  <p>I tried two different ways to get my data:</p>
  
  <p>A simple one:</p>
  
  <pre><code>for company_name in data:
       search = company_name
       results = 1
       page = requests.get("https://www.google.com/search?q={}&amp;num={}".format(search, results))
  
       soup = BeautifulSoup(page.content, "html5lib")
  </code></pre>
  
  <p>and a more complex on:</p>
  
  <pre><code>for company_name in data:
      search = company_name
      results = 1
  
      s = requests.Session()
      retries = Retry(total=3, backoff_factor=0.5)
      s.mount('http://', HTTPAdapter(max_retries=retries))
      s.mount('https://', HTTPAdapter(max_retries=retries))
      page = s.get("https://www.google.com/search?q={}&amp;num={}".format(search, results))
      #time.sleep(.600)
  
      soup = BeautifulSoup(page.content, "html5lib")
  </code></pre>
  
  <p>But I'm getting the same mistake over and over. Is there a way I could overcome this issue? thanks!</p>