title: Scrapy retry or redirect middleware
body: <p>While crawling through a site with scrapy, I get redirected to a user-blocked page about 1/5th of the time.  I lose the pages that I get redirected <em>from</em> when that happens.  I don't know which middleware to use or what settings to use in that middleware, but I want this:</p>
  
  <p><code>DEBUG: Redirecting (302) to (GET http://domain.com/foo.aspx) from (GET http://domain.com/bar.htm)</code></p>
  
  <p>To NOT drop bar.htm.  I end up with no data from bar.htm when the scraper's done, but I'm rotating proxies, so if it tries bar.htm again (maybe a few more times), I should get it.  How do I set the number of tries for that?</p>
  
  <p>If it matters, I'm only allowing the crawler to use a very specific starting url and then only follow "next page" links, so it should go in order through a small number of pages - hence why I need it to either retry, e.g., page 34, or come back to it later.  Scrapy documentation says it should retry 20 times by default, but I don't see it retrying at all.  Also if it helps: All redirects go to the same page (a "go away" page, the foo.com above) - is there a way to tell Scrapy that that particular page "doesn't count" and if it's getting redirected there, to keep retrying?  I saw something in the downloader middleware referring to particular http codes in a list - can I add 302 to the "always keep trying this" list somehow?</p>