How to fake javascript enabled in Python requests/beautifulsoup
Question: I'm trying to crawl a website which return an error message that your js is
disabled and you might be a bot. I tried to see same behaviour in web browser
and yes the same response, however if JavaScript is enabled it will not affect
the original response, I mean original response is not dependent on JS.

So I was thinking if I can tell the web/http server that my JS is enabled and
I'm not a BOT. is this possible in Python requests library, or any other
python library for that matter?

And yeah I've set the `User-Agent` header, even all other headers, like
`host`, `language`, `connection`, etc


Answer: If the site is just checking whether javascript can be executed or not through
executing some js, use selenium to get the page, and then use BeautifulSoup to
parse the page that selenium got.

    
    
    from bs4 import BeautifulSoup
    from selenium import webdriver
    driver = webdriver.Firefox()
    driver.get('http://your-site/url')
    
    html = driver.page_source
    soup = BeautifulSoup(html)
    ...
    

