title: delaying the python bs4 html loading to load the html of the page of the website to extract it
body: <p>m using this code for scrapping some data from the link <a href="https://website.grader.com/results/www.dubizzle.com" rel="nofollow noreferrer">https://website.grader.com/results/www.dubizzle.com</a></p>
  
  <p>The code is as below</p>
  
  <pre><code>#!/usr/bin/python
  import urllib
  from bs4 import BeautifulSoup
  from dateutil.parser import parse
  from datetime import timedelta
  
  import MySQLdb
  import re
  import pdb
  import sys
  import string
  
  def getting_urls_of_all_pages(): 
      url_rent_flat='https://website.grader.com/results/dubizzle.com'
      every_property_in_a_page_data_extraction(url_rent_flat) 
  
  
  def every_property_in_a_page_data_extraction(url):
  
      htmlfile=urllib.urlopen(url).read()
      soup=BeautifulSoup(htmlfile)
  
      print soup
  
      Sizeofweb=""
      try:
  
          Sizeofweb= soup.find('span', {'data-reactid': ".0.0.3.0.0.3.$0.1.1.0"}).text
          print Sizeofweb.get_text().encode("utf-8")
  
      except StandardError as e:
          error="Error was {0}".format(e)
          print error
  
  getting_urls_of_all_pages()
  </code></pre>
  
  <p>The part of the html which i am extracting is as below</p>
  
  <p>Snap: <a href="https://www.dropbox.com/s/7dwbaiyizwa36m6/5.PNG?dl=0" rel="nofollow noreferrer">https://www.dropbox.com/s/7dwbaiyizwa36m6/5.PNG?dl=0</a></p>
  
  <p>Code:</p>
  
  <pre><code>&lt;div class="result-value" data-reactid=".0.0.3.0.0.3.$0.1.1"&gt;
  &lt;span data-reactid=".0.0.3.0.0.3.$0.1.1.0"&gt;1.1&lt;/span&gt;
  &lt;span class="result-value-unit" data-reactid=".0.0.3.0.0.3.$0.1.1.1"&gt;MB&lt;/span&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>Problem: Problem is that the website takes around 10-15 seconds to load the html source file which has the tags which i want to extract as mentioned in the code.</p>
  
  <p>When the code uses the line htmlfile=urllib.urlopen(url).read() to load the html of the page, I think it loads html of preload of the link which is there before 10-15 seconds.</p>
  
  <p>How can i make a pause in the code and let it load the data after 15+ seconds so the right html with the tags i want to extract loads in the program?</p>
  
  <p>someone recommended me to use selenium. Here is the code but not sure whether if can be integrated in my code and serves the purposes</p>
  
  <pre><code>from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  
  driver = webdriver.Firefox()
  driver.get(baseurl)
  </code></pre>