title: How to properly use Rules, restrict_xpaths to crawl and parse URLs with scrapy?
body: <p>I am trying to program a crawl spider to crawl RSS feeds of a website and then parsing the meta tags of the article.</p>
  
  <p>The first RSS page is a page that displays the RSS categories. I managed to extract the link because the  tag is in a  tag. It looks like this:</p>
  
  <pre><code>        &lt;tr&gt;
             &lt;td class="xmlLink"&gt;
               &lt;a href="http://feeds.example.com/subject1"&gt;subject1&lt;/a&gt;
             &lt;/td&gt;   
          &lt;/tr&gt;
          &lt;tr&gt;
             &lt;td class="xmlLink"&gt;
               &lt;a href="http://feeds.example.com/subject2"&gt;subject2&lt;/a&gt;
             &lt;/td&gt;
          &lt;/tr&gt;
  </code></pre>
  
  <p>Once you click that link it brings you the the articles for that RSS category that looks like this:</p>
  
  <pre><code>   &lt;li class="regularitem"&gt;
      &lt;h4 class="itemtitle"&gt;
          &lt;a href="http://example.com/article1"&gt;article1&lt;/a&gt;
      &lt;/h4&gt;
    &lt;/li&gt;
    &lt;li class="regularitem"&gt;
       &lt;h4 class="itemtitle"&gt;
          &lt;a href="http://example.com/article2"&gt;article2&lt;/a&gt;
       &lt;/h4&gt;
    &lt;/li&gt;
  </code></pre>
  
  <p>As You can see I can get the link with xpath again if I use the  tag
  I want my crawler to go to the link inside that tag and parse the meta tags for me.</p>
  
  <p>Here is my crawler code:</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from tutorial.items import exampleItem
  
  
  class MetaCrawl(CrawlSpider):
      name = 'metaspider'
      start_urls = ['http://example.com/tools/rss'] # urls from which the spider will start crawling
      rules = [Rule(SgmlLinkExtractor(restrict_xpaths=('//td[@class="xmlLink"]')), follow=True),
          Rule(SgmlLinkExtractor(restrict_xpaths=('//h4[@class="itemtitle"]')), callback='parse_articles')]
  
      def parse_articles(self, response):
          hxs = HtmlXPathSelector(response)
          meta = hxs.select('//meta')
          items = []
          for m in meta:
             item = exampleItem()
             item['link'] = response.url
             item['meta_name'] =m.select('@name').extract()
             item['meta_value'] = m.select('@content').extract()
             items.append(item)
          return items
  </code></pre>
  
  <p>However this is the output when I run the crawler:</p>
  
  <pre><code>DEBUG: Crawled (200) &lt;GET http://http://feeds.example.com/subject1&gt; (referer: http://example.com/tools/rss)
  DEBUG: Crawled (200) &lt;GET http://http://feeds.example.com/subject2&gt; (referer: http://example.com/tools/rss)
  </code></pre>
  
  <p>What am I doing wrong here? I've been reading the documentation over and over again but I feel like I keep overlooking something. Any help would be appreciated.</p>
  
  <p><strong>EDIT:</strong> Added: items.append(item) . Had forgotten it in original post.
  <strong>EDIT:</strong> : I've tried this as well and it resulted in the same output:</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from reuters.items import exampleItem
  from scrapy.http import Request
  
  class MetaCrawl(CrawlSpider):
      name = 'metaspider'
      start_urls = ['http://example.com/tools/rss'] # urls from which the spider will start crawling
      rules = [Rule(SgmlLinkExtractor(allow=[r'.*',], restrict_xpaths=('//td[@class="xmlLink"]')), follow=True),
               Rule(SgmlLinkExtractor(allow=[r'.*'], restrict_xpaths=('//h4[@class="itemtitle"]')),follow=True),]
  
  
      def parse(self, response):       
          hxs = HtmlXPathSelector(response)
          meta = hxs.select('//td[@class="xmlLink"]/a/@href')
          for m in meta:
              yield Request(m.extract(), callback = self.parse_link)
  
  
      def parse_link(self, response):       
          hxs = HtmlXPathSelector(response)
          meta = hxs.select('//h4[@class="itemtitle"]/a/@href')
          for m in meta:
              yield Request(m.extract(), callback = self.parse_again)    
  
      def parse_again(self, response):
          hxs = HtmlXPathSelector(response)
          meta = hxs.select('//meta')
          items = []
          for m in meta:
              item = exampleItem()
              item['link'] = response.url
              item['meta_name'] = m.select('@name').extract()
              item['meta_value'] = m.select('@content').extract()
              items.append(item)
          return items
  </code></pre>