title: Saving Web Crawling Results (Scrapy)
body: <p>I've written a spider that seems to be functioning properly, but I'm not sure how to save the data it is collecting.</p>
  <p>The spider starts out at <a href="http://www.thescienceforum.com/" rel="nofollow noreferrer">TheScienceForum</a>, grabs the main forum pages and makes an item for each. It then proceeds to go through all of the individual forum pages (passing the items along with it), adding the title of each thread to the matching forum item. The code is as follow:</p>
  <pre><code>from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.http import Request
  
  from individualProject.items import ProjectItem
  
  class TheScienceForum(BaseSpider):
      name = &quot;TheScienceForum.com&quot;
      allowed_domains = [&quot;www.thescienceforum.com&quot;]
      start_urls = [&quot;http://www.thescienceforum.com&quot;]
      
      def parse(self, response):
          Sel = HtmlXPathSelector(response)
          forumNames = Sel.select('//h2[@class=&quot;forumtitle&quot;]/a/text()').extract()
          items = []
          for forumName in forumNames:
              item = ProjectItem()
              item['name'] = forumName
              items.append(item)
          
  
          forums = Sel.select('//h2[@class=&quot;forumtitle&quot;]/a/@href').extract()
          itemDict = {}
          itemDict['items'] = items
          for forum in forums:
              yield Request(url=forum,meta=itemDict,callback=self.addThreadNames)  
  
      def addThreadNames(self, response):
          items = response.meta['items']
          Sel = HtmlXPathSelector(response)
          currentForum = Sel.select('//h1/span[@class=&quot;forumtitle&quot;]/text()').extract()
          for item in items:
              if currentForum==item['name']:
                  item['thread'] += Sel.select('//h3[@class=&quot;threadtitle&quot;]/a/text()').extract()
          self.log(items)
  
  
          itemDict = {}
          itemDict['items'] = items
          threadPageNavs = Sel.select('//span[@class=&quot;prev_next&quot;]/a[@rel=&quot;next&quot;]/@href').extract()
          for threadPageNav in threadPageNavs:  
              yield Request(url=threadPageNav,meta=itemDict,callback=self.addThreadNames)
  </code></pre>
  <p>It seems that because I never simply return the object (only new requests are yielded) that the data never persists anywhere. I've tried using the following JSON pipeline:</p>
  <pre><code>class JsonWriterPipeline(object):
  
      def __init__(self):
          self.file = open('items.jl', 'wb')
  
      def process_item(self, item, spider):
          line = json.dumps(dict(item)) + &quot;\n&quot;
          self.file.write(line)
          return item
  </code></pre>
  <p>And also running the spider with the following:</p>
  <pre><code>scrapy crawl TheScienceForum.com -o items.json -t json
  </code></pre>
  <p>but so far nothing is working. Where might I be going wrong?</p>
  <p>Any ideas or positive criticism is warmly welcomed.</p>