title: python urllib2 - wait for page to finish loading/redirecting before scraping?
body: <p>I'm learning to make web scrapers and want to scrape TripAdvisor for a personal project, grabbing the html using urllib2.
  However, I'm running into a problem where, using the code below, the html I get back is not correct as the page seems to take a second to redirect (you can verify this by visiting the url) - instead I get the code from the page that initially briefly appears.</p>
  
  <p>Is there some behavior or parameter to set to make sure the page has completely finished loading/redirecting before getting the website content?</p>
  
  <pre><code>import urllib2
  from bs4 import BeautifulSoup
  
  bostonPage = urllib2.urlopen("http://www.tripadvisor.com/HACSearch?geo=34438#02,1342106684473,rad:S0,sponsors:ABEST_WESTERN,style:Szff_6")
  soup = BeautifulSoup(bostonPage)
  print soup.prettify()
  </code></pre>
  
  <p>Edit: The answer is thorough, however, in the end what solved my problem was this:
  <a href="https://stackoverflow.com/a/3210737/1157283">https://stackoverflow.com/a/3210737/1157283</a></p>