text: .
  What follows is a working example of how to do that, using the testspiders project as example. Remember that Scrapy is built on top of the Twisted asynchronous networking library, so you need run it inside the Twisted reactor.
  from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy.settings import Settings from scrapy import log from testspiders.spiders.followall import FollowAllSpider spider = FollowAllSpider(domain='scrapinghub.com') crawler = Crawler(Settings()) crawler.configure() crawler.crawl(spider) crawler.start() log.start() reactor.run() # the script will block here() kind of sites:
  - rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)
  - disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour
  - use download delays (2 or higher). See DOWNLOAD_DELAY setting.
  - if possible, use Google cache to fetch pages, instead of hitting the sites directly
  - use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh
  If you are still unable to prevent your bot getting banned, consider contacting commercial support.