title: What is the simplest way to programatically start a crawler in Scrapy >= 0.14
body: <p>I want to start a crawler in Scrapy from a Python module. I want to essentially mimic the essence of <code>$ scrapy crawl my_crawler -a some_arg=value -L DEBUG</code></p>
  
  <p>I have the following things in place:</p>
  
  <ul>
  <li>a settings.py file for the project</li>
  <li>items and pipelines</li>
  <li>a crawler class which extends BaseSpider and requires arguments upon initialisation.</li>
  </ul>
  
  <p>I can quite happily run my project using the <code>scrapy</code> command as specified above, however I'm writing integration tests and I want to programatically:</p>
  
  <ul>
  <li>launch the crawl using the settings in <code>settings.py</code> and the crawler that has the <code>my_crawler</code> name attribute (I can instantiate this class easily from my test module.</li>
  <li>I want all the pipelines and middleware to be used as per the specification in <code>settings.py</code>.</li>
  <li>I'm quite happy for the process to be blocked until the crawler has finished. The pipelines dump things in a DB and it's the contents of the DB I'll be inspecting after the crawl is done to satisfy my tests.</li>
  </ul>
  
  <p>So, can anyone help me? I've seen some examples on the net but they are either hacks for multiple spiders, or getting around <code>Twisted's</code> blocking nature, or don't work with Scrapy 0.14 or above. I just need something real simple. :-)</p>