title: How to Scrape a search form in scrapy
body: <p>I am trying to use scrapy to scrape a website that has a search form but it's hard to do concurrent requests given how it works:
  when you perform a search, it kind of creates a session around the search and you cannot have several searches at the same time, otherwise you cannot access the results of the previous searches anymore.</p>
  
  <p>For example:</p>
  
  <p>Starting a search with criteria A, it redirects to a page with an ID and the details of each result reuse this ID:</p>
  
  <pre><code>list.do?anchor=51edc79.0
  details.do?anchor=51edc79.0.0
  details.do?anchor=51edc79.0.1
  </code></pre>
  
  <p>If I start a new search with criteria B, it's a new ID:</p>
  
  <pre><code>list.do?anchor=5200109.0
  details.do?anchor=5200109.0.0
  details.do?anchor=5200109.0.1   
  </code></pre>
  
  <p>But now the results of the 1st search are not accessible anymore:</p>
  
  <pre><code>details.do?anchor=51edc79.0.2 =&gt; cannot find
  details.do?anchor=51edc79.0.3 =&gt; cannot find
  </code></pre>
  
  <p>I could do only one request at a time to make sure the 2nd search won't start until all the results of the 1st search are fetched but I'd still like to make concurrent requests to get the results of one search more quickly.
  So the idea would be to not start the 2nd search until all the results of a search are fetched.</p>
  
  <p>How would you do that in scrapy?</p>
  
  <p>When I allowed multiple requests (<strong><code>CONCURRENT_REQUESTS</code></strong>) and use priorities, the 2nd search started before all the pages from the first search were done.</p>
  
  <p>Thanks</p>