title: Scrapy RSS Scraper
body: <p>I am trying to scrape an RSS feed from Yahoo ( their Open Company RSS Feed | <a href="https://developer.yahoo.com/finance/company.html" rel="nofollow noreferrer">https://developer.yahoo.com/finance/company.html</a> ) </p>
  
  <p>I am trying to scrape the following URL: <a href="https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX" rel="nofollow noreferrer">https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX</a></p>
  
  <p>For some reason my spider isn't functioning, and I think it might have to do with either the XPath that's generated, if not, defining parse_item might have some issues.</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider
  from YahooScrape.items import YahooScrapeItem
  
  class Spider(CrawlSpider):
      name= "YahooScrape"
      allowed_domains = ["yahoo.com"]
      start_urls = ('https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX',)
  
     def parse_item(self, response):
          self.logger.info('Hi, this is an item page! %s', response.url)
          item = EmperyscraperItem()
          item['title'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()                #define XPath for title
          item['link'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()                 #define XPath for link
          item['description'] = response.xpath('//*[@id="collapsible"]/div[1]/div[2]/span',).extract()          #define XPath for description
          return item
  </code></pre>
  
  <p>What could be the issue with the code? If not, what are the proper XPath directions to extract the title, desc, and link. I'm new to Scrapy and just need some help figuring it out!</p>
  
  <p><strong>Edit:</strong> I've updated my spider and converted it into an XMLFeedSpider as shown below: </p>
  
  <pre><code>import scrapy
  
  from scrapy.spiders import XMLFeedSpider
  from YahooScrape.items import YahooScrapeItem
  
  class Spider(XMLFeedSpider):
      name = "YahooScrape"
      allowed_domains = ["yahoo.com"]
      start_urls = ('https://feeds.finance.yahoo.com/rss/2.0/headline?s=BPMX')    #Crawl BPMX
      itertag = 'item'
  
      def parse_node(self, response, node):
          self.logger.info('Hi, this is a &lt;%s&gt; node!: %s', self.itertag, ''.join(node.extract()))
  
          item = YahooScrapeItem()
          item['title'] = node.xpath('item/title/text()',).extract()                #define XPath for title
          item['link'] = node.xpath('item/link/text()').extract()
          item['pubDate'] = node.xpath('item/link/pubDate/text()').extract()
          item['description'] = node.xpath('item/category/text()').extract()                #define XPath for description
          return item
  
  #Yahoo RSS feeds http://finance.yahoo.com/rss/headline?s=BPMX,APPL
  </code></pre>
  
  <p>Now I'm getting the following error:</p>
  
  <pre><code>2017-06-13 11:25:57 [scrapy.core.engine] ERROR: Error while obtaining start requests
  </code></pre>
  
  <p>Any idea why the error has occurred? My HTML path looks correct.</p>