title: Scraping incomplete tags using python and BeautifulSoup
body: <p>Attempting to scrape this site <a href="http://www.thepetitionsite.com/takeaction/736/152/031/" rel="nofollow noreferrer">url</a> using urllib2, mechanize and BeautifulSoup. When rendered through a browser and viewed using FireBug we can see the div tags with class progress__supporters and progress_goal as shown below.</p>
  
  <p><a href="https://i.stack.imgur.com/fQD3x.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fQD3x.png" alt="Fire Bug View"></a></p>
  
  <p>But when we use urllib2.open/mechanize/selenium to view the webpage, these tags are not available. Below is the sample code </p>
  
  <pre><code>    def parse_petition(self,petition):
          mb = mechanize.Browser()
          mb.addheaders = [('User-agent',
                        'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
          mb.set_handle_robots(False)
          data = mb.open(petition,timeout=50).read()
          soup = BeautifulSoup(data)
          print soup.find('progress')
  </code></pre>
  
  <p>We get the following output using the above code. </p>
  
  <pre><code>&lt;div class='progress'&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>One reason could be because the div tags with class 'progress__goal' and 'progress__support' are incomplete. How could I read these pages as exactly as the source and scrape the values using BeautifulSoup?</p>