title: Blocked while scraping goodreads.com
body: <p>I am trying to scrape a large sample (100k+) of the books available at "<a href="https://www.goodreads.com/book/show/" rel="nofollow noreferrer">https://www.goodreads.com/book/show/</a>", but I get continuously blocked.
  So far I have tried to implement the following solutions in my code:</p>
  
  <ul>
  <li><p>Check the robots.txt to find what sites/elements are not accessible</p></li>
  <li><p>Specify a header or multiple headers that randomly change</p></li>
  <li><p>Use multiple working proxies to avoid being blocked</p></li>
  <li><p>Set a delay up to 20 seconds between each scraping iteration that uses 10 simultaneous threads</p></li>
  </ul>
  
  <p>Here is a simplified version of the code that gets blocked while trying to scrape only the title and author of the book, without using multiple simultaneous threads:</p>
  
  <pre><code>import requests
  from lxml import html
  import random
  
  proxies_list = ["http://89.71.193.86:8080", "http://178.77.206.21:59298", "http://79.106.37.70:48550",
                  "http://41.190.128.82:47131", "http://159.224.109.140:38543", "http://94.28.90.214:37641",
                  "http://46.10.241.140:53281", "http://82.147.120.30:56281", "http://41.215.32.86:55561"]
  proxies = {"http": random.choice(proxies_list)}
  
  # real header
  # headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}
  
  # multiple headers
  headers_list = ['Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36',
                  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.38 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1623.0 Safari/537.36',
                  'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36']
  headers = {"user-agent": random.choice(headers_list)}
  
  first_url = 1
  last_url = 10000     # Last book is 8,630,000
  sleep_time = 20
  
  for book_reference_number in range(first_url, last_url):
      try:
          goodreads_html = requests.get("https://www.goodreads.com/book/show/" + str(book_reference_number), timeout=5, headers=headers, proxies=proxies)
          doc = html.fromstring(goodreads_html.text)
          book_title = doc.xpath('//div[@id="topcol"]//h1[@id="bookTitle"]')[0].text.strip(", \t\n\r")
          try:
              author_name = doc.xpath('//div[@id="topcol"]//a[@class="authorName"]//span')[0].text.strip(", \t\n\r")
          except:
              author_name = ""
          time.sleep(sleep_time)
          print(str(book_reference_number), book_title, author_name)
      except:
          print(str(book_reference_number) + " cannot be scraped.")
          pass
  </code></pre>