During my career I see the battle between website/web app owners and bots/scrapers/crawlers writers. I thought this battle can’t be won. But about 6 months ago I joined it and I think now I have [almost] deadly weapon.
Selenium Webdriver is my choice.
Probably, you heard or used it before. It’s the most popular tool for the functional tests (also known as end-to-end tests), and projects like saucelabs.com can make these tests very easy to implement and run.
So, you wrote a sequence of steps for scraping some website. Awesome! But what step should be next? Of course you can just run it manually on your computer, but what if you need to create some sort of service or even platform based on it? Yes, it’s possible!
now you can run your application! Everything should just work, including screenshots (useful for debugging).
val defaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
It takes URL to download, path where it should save the file, set of cookies and optional user agent header. Of course you can pass and add more headers if you need.
CAPTCHA. But there are a lot of tools that can help with recognition, so it can’t be really serious protection.
Heuristic methods. For example, Google AdWords/AdSense system is able to detect bots by tracking mouse moves, scrolls, timings, etc. I believe it’s very complicated and very expensive technology, but it exists.