title: python scrape flight and price data from skyscanner
body: <p>I am trying to get the price data from the following url. However I can only seem to get the text from 'div's down to a certain level, here is my code:</p>
  
  <pre><code>from selenium import webdriver
  from bs4 import BeautifulSoup
  
  def scrape_flight_prices(URL):
  
      browser = webdriver.PhantomJS()
      # PARSE THE HTML
      browser.get(URL)
      soup = BeautifulSoup(browser.page_source, "lxml")
      page_divs = soup.findAll("div", attrs={'id':'app-root'}) 
      for p in page_divs:
          print(p)
  
  if __name__ == '__main__':
    URL1="https://www.skyscanner.net/transport/flights/brs/gnb/190216/190223/?adults=1&amp;children=0&amp;adultsv2=1&amp;childrenv2=&amp;infants=0&amp;cabinclass=economy&amp;rtn=1&amp;preferdirects=false&amp;outboundaltsenabled=false&amp;inboundaltsenabled=false&amp;ref=home#results"
  </code></pre>
  
  <p>And here is the output:</p>
  
  <pre><code>&lt;div id="app-root"&gt;
  &lt;section class="day-content state-loading state-no-results" id="daysection"&gt;
  &lt;div class="day-searching"&gt;
  &lt;div class="hot-spinner medium"&gt;&lt;/div&gt;
  &lt;div class="day-searching-message"&gt;Searching&lt;/div&gt;
  &lt;/div&gt;
  &lt;/section&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>The section of html I want to scrape from looks like this:</p>
  
  <pre><code>https://www.skyscanner.net/transport/flights/brs/gnb/190216/190223/?adults=1&amp;children=0&amp;adultsv2=1&amp;childrenv2=&amp;infants=0&amp;cabinclass=economy&amp;rtn=1&amp;preferdirects=false&amp;outboundaltsenabled=false&amp;inboundaltsenabled=false&amp;ref=home#results
  </code></pre>
  
  <p>However when I try and scrape with the following code:</p>
  
  <pre><code>prices = soup.findAll("a", attrs={'target':"_blank", "data-e2e":"itinerary-price", "class":"CTASection__price-2bc7h price"})  
  for p in prices:
      print(p)
  </code></pre>
  
  <p>It prints nothing! I suspect a js script is running something to generate the rest of the the code and/or data? Can anyone help me extract the data? Specifically I am trying to get the price, flight times, airline name etc but if beautiful soup is not printing the relevant html from the page then I'm not sure how else to get it?</p>
  
  <p>Would appreciate any pointers! Many thanks in advance!</p>