title: Scrapy is following and scraping non-allowed links
body: <p>I have a CrawlSpider set up to following certain links and scrape a news magazine where the links to each issue follow the following URL scheme:</p>
  
  <p><a href="http://example.com/YYYY/DDDD/index.htm">http://example.com/YYYY/DDDD/index.htm</a> where YYYY is the year and DDDD is the three or four digit issue number.</p>
  
  <p>I only want issues 928 onwards, and have my rules below. I don't have any problem connecting to the site, crawling links, or extracting items (so I didn't include the rest of my code). The spider seems determined to follow non-allowed links. It is trying to scrape issues 377, 398, and more, and follows the "culture.htm" and "feature.htm" links. This throws a lot of errors and isn't terribly important but it requires a lot of cleaning of the data. Any suggestions as to what is going wrong?</p>
  
  <pre><code>class crawlerNameSpider(CrawlSpider):
  name = 'crawler'
  allowed_domains = ["example.com"]
  start_urls = ["http://example.com/issues.htm"]
  
  rules = (
          Rule(SgmlLinkExtractor(allow = ('\d\d\d\d/(92[8-9]|9[3-9][0-9]|\d\d\d\d)/index\.htm', )), follow = True),
          Rule(SgmlLinkExtractor(allow = ('fr[0-9].htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('eg[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('ec[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('op[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('sc[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('re[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(allow = ('in[0-9]*.htm', )), callback = 'parse_item'),
          Rule(SgmlLinkExtractor(deny = ('culture.htm', )), ),
          Rule(SgmlLinkExtractor(deny = ('feature.htm', )), ),
      )
  </code></pre>
  
  <p>EDIT: I fixed this using a much simpler regex fot 2009, 2010, 2011, but I am still curious why the above doesn't work if anyone has any suggestions.</p>