title: How to stop redirect and stay on the page to scrape the data (python3, beautifulsoup,urllib)
body: <p>I am trying to scrape the review from a website here: <a href="http://movie.douban.com/subject/2303845/comments" rel="nofollow">http://movie.douban.com/subject/2303845/comments</a></p>
  
  <p>And apparently it keeps on redirecting itself back to the main page. I would like to find a way to stop the redirecting and stay on the same page to get the part of html that has div with comment as its class. </p>
  
  <p>Keywords: Python3, Beautifulsoup, urllib</p>
  
  <p>I have read other posts and below is what I have tried: </p>
  
  <pre><code>#!/usr/bin/env python
  # -*- coding: utf-8 -*- 
  import sys
  from bs4 import BeautifulSoup
  from urllib.request import urlopen
  
  sys.setrecursionlimit(30000)
  
  try:
      import urllib.request as urllib2
      import urllib.request as urllib
  except ImportError:
      import urllib2
  
  class NoRedirectHandler(urllib2.HTTPRedirectHandler):
      def http_error_302(self, req, fp, code, msg, headers):
          infourl = urllib.addinfourl(fp, headers, req.get_full_url())
          infourl.status = code
          infourl.code = code
          return infourl
      http_error_300 = http_error_302
      http_error_301 = http_error_302
      http_error_303 = http_error_302
      http_error_307 = http_error_302
  
  opener = urllib2.build_opener(NoRedirectHandler())
  urllib2.install_opener(opener)
  response = urllib2.urlopen('http://movie.douban.com/subject/2303845/comments')
  soup = BeautifulSoup(response, "html.parser")
  comments = soup.findAll('div', {'class': 'comment'})
  print (comments)
  
  if response.code in (300, 301, 302, 303, 307):
      print ('redirect')
  </code></pre>
  
  <p>For some reason, I think it is still not getting the data and I suspects it never stops redirecting to the other page.</p>