the url you need to pass to requests get(howeversometimes this isn' so easy to find or perhaps the website you want your program to navigate requires you to log in first the selenium module will give your programs the power to perform such sophisticated tasks controlling the browser with the selenium module the selenium module lets python directly control the browser by programmatically clicking links and filling in login informationalmost as though there were human user interacting with the page using seleniumyou can interact with web pages in much more advanced way than with requests and bs but because it launches web browserit is bit slower and hard to run in the background ifsayyou just need to download some files from the web stillif you need to interact with web page in way thatsaydepends on the javascript code that updates the pageyou'll need to use selenium instead of requests that' because major ecommerce websites such as amazon almost certainly have software systems to recognize traffic that they suspect is script harvesting their info or signing up for multiple free accounts these sites may refuse to serve pages to you after whilebreaking any scripts you've made the selenium module is much more likely to function on these sites long-term than requests major "tellto websites that you're using script is the user-agent stringwhich identifies the web browser and is included in all http requests for examplethe user-agent string for the requests module is something like 'python-requests/you can visit site such as www whatsmyua infoto see your user-agent string using seleniumyou're much more likely to "pass for humanbecause not only is selenium' useragent is the same as regular browser (for instance'mozilla/ (windows nt win rv: gecko/ firefox/ ')but it has the same traffic patternsa selenium-controlled browser will download imagesadvertisementscookiesand privacy-invading trackers just like regular browser howeverselenium can still be detected by websitesand major ticketing and ecommerce websites often block browsers controlled by selenium to prevent web scraping of their pages starting selenium-controlled browser the following examples will show you how to control firefox' web browser if you don' already have firefoxyou can download it for free from getfirefox comyou can install selenium by running pip install --user selenium from command line terminal more information is available in appendix importing the modules for selenium is slightly tricky instead of import seleniumyou need to run from selenium import webdriver (the exact reason why the selenium module is set up this way is beyond the scope of this book web scraping