title: How to avoid being banned while scraping data from a login based site?
body: <p>I'm trying to create a script using which I can parse few fields from a website without getting blocked. The site I wish to get data from requires credentials to access it's content. If it were not for login thing, I could have bypassed the rate limit using rotation of proxies.</p>
  <p>As I'm scraping content from a login based site, I'm trying to figure out any way to avoid being banned by that site while scraping data from there. <code>To be specific, my script currently can fetch content from that site flawlessly but my ip address gets banned along the way if I keep on scraping</code>.</p>
  <p>I've written so far (<em><strong>consider the following site address to be a placeholder</strong></em>):</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  url = &quot;https://stackoverflow.com/users/login?ssrc=head&amp;returnurl=https%3a%2f%2fstackoverflow.com%2f&quot;
  
  with requests.Session() as s:
      s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
      req = s.get(url)
  
      payload = {
          &quot;fkey&quot;: BeautifulSoup(req.text,&quot;lxml&quot;).select_one(&quot;[name='fkey']&quot;)[&quot;value&quot;],
          &quot;email&quot;: &quot;some email&quot;,
          &quot;password&quot;: &quot;some password&quot;,
      }
      
      res = s.post(url,data=payload)
      soup = BeautifulSoup(res.text,&quot;lxml&quot;)
      for post_title in soup.select(&quot;.summary &gt; h3 &gt; a.question-hyperlink&quot;):
          print(post_title.text)
  </code></pre>
  <blockquote>
  <p>How can I avoid being banned while scraping data from a login based site?</p>
  </blockquote>