title: When and how should use multiple spiders in one Scrapy project
body: <p>I am using <code>Scrapy</code>, it is great!  so fast to build a crawler.  with the number of web sites are increasing,  need to create new spiders, but these web sits are the same type,
  all these spiders use  same items, pipelines, parsing process</p>
  
  <p>the contents of the project directory:</p>
  
  <pre><code>test/
  ├── scrapy.cfg
  └── test
      ├── __init__.py
      ├── items.py
      ├── mybasespider.py
      ├── pipelines.py
      ├── settings.py
      ├── spider1_settings.py
      ├── spider2_settings.py
      └── spiders
          ├── __init__.py
          ├── spider1.py
          └── spider2.py
  </code></pre>
  
  <p>To reduce source code redundancy, <code>mybasespider.py</code> has a base spider <code>MyBaseSpider</code>, 95% source code are in it,  all other spiders inherited from it, if a spider has some special things, override some 
  <code>class methods</code>, normally only need to add several lines source code to create a new spider</p>
  
  <p>Place all common settings in <code>settings.py</code>,   one spider's special settings are in <code>[spider name]_settings.py</code>, such as:</p>
  
  <p>the special settings of <code>spider1</code> in <code>spider1_settings.py</code>:</p>
  
  <pre><code>from settings import *
  
  LOG_FILE = 'spider1.log'
  LOG_LEVEL = 'INFO'
  JOBDIR = 'spider1-job'
  START_URLS = [
      'http://test1.com/',
  ]
  </code></pre>
  
  <p>the special settings of <code>spider2</code> in <code>spider2_settings.py</code>:</p>
  
  <pre><code>from settings import *
  
  LOG_FILE = 'spider2.log'
  LOG_LEVEL = 'DEBUG'
  JOBDIR = 'spider2-job'
  START_URLS = [
      'http://test2.com/',
  ]
  </code></pre>
  
  <p><code>Scrapy</code> uses <code>LOG_FILE</code>, <code>LOG_LEVEL</code>, <code>JOBDIR</code> before launching a spider;</p>
  
  <p>All urls in <code>START_URLS</code> are filled into <code>MyBaseSpider.start_urls</code>, different spider has different contents, but the name <code>START_URLS</code> used in the base spider <code>MyBaseSpider</code> isn't changed.</p>
  
  <p>the contents of the <code>scrapy.cfg</code>:</p>
  
  <pre><code>[settings]
  default = test.settings
  spider1 = spider1.settings
  spider2 = spider2.settings
  
  [deploy]
  url = http://localhost:6800/
  project = test
  </code></pre>
  
  <p>To run a spider, such as <code>spider1</code>:</p>
  
  <ol>
  <li><p><code>export SCRAPY_PROJECT=spider1</code></p></li>
  <li><p><code>scrapy crawl spider1</code></p></li>
  </ol>
  
  <p>But this way can't be used to run spiders in <code>scrapyd</code>. <code>scrapyd-deploy</code> command always uses <code>'default'</code> project name in <code>scrapy.cfg</code> 'settings' section to build an <code>egg file</code> and deploys it to <code>scrapyd</code></p>
  
  <p>Have several questions:</p>
  
  <ol>
  <li><p>Is this the way to use multiple spiders in one project if I don't create a project per spider? Are there any better ways?</p></li>
  <li><p>How to separate a spider's special settings as above which can run in <code>scrapyd</code> and reduce source code redundancy</p></li>
  <li><p>If all spiders use a same <code>JOBDIR</code>, is it safe to run all spiders concurrently? Is the persistent spider state corrupted?</p></li>
  </ol>
  
  <p>Any insights would be greatly appreciated.</p>