title: Beautifulsoup Webscraping: How to get information with javascript?
body: <p>I am attempting to webscrape a specific page from Choice Hotel's website (specifically <a href="https://www.choicehotels.com/tennessee/nashville/hotels" rel="nofollow noreferrer">https://www.choicehotels.com/tennessee/nashville/hotels</a> ) to create a list of all choice hotels in Nashville, TN. When I open the page and open up my developer's tools, I can see the information I'm looking for under <code>&lt;div class="list"&gt;</code>, however, when I attempt to scrape the site, I cannot find this tag. I can't seem to find anything deeper than <code>&lt;div class="animate-fade z-index-90"&gt;</code>, which any tags on a deeper level than that just return "None". However, I do see a lot of Javascript towards the bottom. I believe this is due to the request not seeing what I see when I open the page in my browser. How might I make my program able to see the tags that I see?</p>
  
  <p>This is how I am attempting the scrape:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import csv
  
  source = request.get("https://www.choicehotels.com/tennessee/nashville/hotels").text
  soup = BeautifulSoup(source, 'lxml')
  list = soup.find('div', class_='list')
  print(list)
  </code></pre>
  
  <p>Is there something I'm not doing, or doing incorrectly?</p>