Title: Why I can't scrape this website even though the html content is visible?
Tags: web-scraping;python-requests-html
Question: This is the link to scrape
I don't know why but I can't scrape the data after the  which contains the entire data in this website.
I used requests_html and beautiful soup but there was no successful results :(
I just gave a basic code.
url = &quot;https://www.bcorporation.net/en-us/find-a-b-corp/search?refinement=countries%3DUnited%20Kingdom&amp;refinement=countries%3DAustralia&amp;refinement=countries%3DBelgium&amp;refinement=countries%3DCanada&amp;refinement=countries%3DChina&amp;refinement=countries%3DDenmark&amp;refinement=countries%3DFrance&amp;refinement=countries%3DGermany&amp;refinement=countries%3DIreland&amp;refinement=countries%3DItaly&amp;refinement=countries%3DNetherlands%20The&amp;refinement=countries%3DMexico&amp;refinement=countries%3DNew%20Zealand&amp;refinement=countries%3DPortugal&amp;refinement=countries%3DSpain&amp;refinement=countries%3DSweden&amp;refinement=countries%3DSwitzerland&amp;refinement=countries%3DUnited%20States&quot;
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246 '
session = HTMLSession()
r = session.get(url, headers=headers)
r.html.render(sleep=50, timeout=50)
content = r.html.find(&quot;ul&quot;)
title = r.html.find(&quot;div[data-testid = company_name]&quot;)
print(title)
Comment: Ohhh I understood David. I would try that

Comment: The website appears to dynamically load much of its data/UI.  Just making a single HTTP request to that URL will only load the *initial* state of the page.  You'll likely either want to reverse-engineer what the page is doing and find where specifically you would get the data you want, or perhaps use more of a "headless browser" tool which would internally process the page's code and update the resulting DOM that you're interacting with.


Here is another answer: looking at the Page the data-testid is company-name not company_name :)
title = r.html.find(&quot;div[data-testid=company-name]&quot;)
worked for me
Comment for this answer: Yeah that was a silly mistake. Btw what David said was the correct one coz I was wondering why the output from the website is just the styles and HTML tags

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2022:00:01/24/22', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<web-scraping><python-requests-html>', 'Title': "Why I can't scrape this website even though the html content is visible?", 'ViewCount': 315, 'id': 70835030, 'user_AboutMe': None, 'user_AccountId': 22389686.0, 'user_DisplayName': 'Cephas Princely', 'user_Reputation': 15.0, 'user_Views': 4.0, 'user_id': 16601977.0}}