title: Scrapy: Rule SgmlLinkExtractor concept
body: <p>Please guide me how to write Rule SgmlLinkExtractor<br>
  I am confused and can't figure out the english documents</p>
  
  <p>I want to crawl the web with many pages<br>
  And the rule is :  </p>
  
  <pre><code> http://abctest.com/list.php?c=&amp;&amp;page=1  
   http://abctest.com/list.php?c=&amp;&amp;page=2  
   http://abctest.com/list.php?c=&amp;&amp;page=3 ...
  </code></pre>
  
  <p>Here is my code:  </p>
  
  <pre><code>from scrapy.selector import Selector
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  import re
  
  class Spider(CrawlSpider):
      name = "find"
      start_urls = ["http://abctest.com/list.php?c=&amp;&amp;page=1",]
      #crawl 2 pages to test if the data is normal  allow=('?c=&amp;&amp;page=/d+')
      rules = [Rule(SgmlLinkExtractor(allow=('?c=&amp;&amp;page=2')),callback='parse_item',follow=True)]
  
  
      #get the page1 item
      def parse(self, response):
          sel = Selector(response)
          sites = sel.css("div#list table tr ")
          for site in sites:
              item = LAItem()
              item['day']  = site.css("  td.date::text ").extract()
              item['URL'] = site.css("  td.subject a::attr(href) ").extract()
           yield item
  
     #get the page2 item    
     def parse_item(self, response):
          sel = Selector(response)
          sites = sel.css("div#list table tr ")
          for site in sites:
              item = LAItem()
              item['day']  = site.css("  td.date::text ").extract()
              item['URL'] = site.css("  td.subject a::attr(href) ").extract()
           yield item   
  </code></pre>