title: Scraping multiple pages, staying independent
body: <p>I want to <strong>scrape</strong> a bunch of pages. Feeding different data pots and are then matched later on.</p>
  <pre><code>[Page1]-Get-PostProcessing-Store-[Pot1]-+
  [Page2]-Get-PostProcessing-Store-[Pot2]-+--Match---[ResultPage]-REST-API
  [Page3]-Get-PostProcessing-Store-[Pot3]-+
  ...
  </code></pre>
  <p>Now I want to be as independent as possible considering the pipeline for each page. Sometimes pages will need JavaScript scraping capabilities, sometimes not. Sometimes I need to also grab images, sometimes only PDFs.</p>
  <p>I did a prototype with one page and Scrapy. I really had the structure and I don't know how to &quot;split&quot; it up that <strong>scraper</strong> and middleware is independent for each page. On the other hand, is lxml enough? How do I handle robots and wait delays to avoid blocking? Does it make sense to add a message queue?</p>
  <p>What is the best way to implement all this? Please be specific! My major problem are structures organizing my code and the tools to use.</p>