title: Create JSON file with urls of all pages with specific keywords in Scrapy
body: <p>I am parsing multiple URLs/pages in scrapy. On each page, it searches for specific keywords and if found, append the URL in Attribute_Dictionary. </p>
  
  <p>As, each URL being parsed, Attribute_Dictionary gets updated every time. Now I want to write a json file of Attribute_Dictionary only ONCE when I am done with parsing of All URLs.</p>
  
  <p>Currently I have written code to dump content in to JSON but it creates JSON file again and again (in a single run) and override the last created json file.<br>
  Desired output is a json file with all urls in Attribure_Dictionary.</p>
  
  <p>Please help. Should I create a global variable to handle all pages that are parsed? If yes then how to do it?</p>
  
  <pre><code>def parse(self, response):
      global parsed_urls
      global domain_urls
      global tld
      global sliced_url
      items = []
      global item
  
  
  
      if ('html' not in response.headers['Content-Type']):
          return
  
      sel = Selector(response)
  
  
      for h3 in sel.xpath('//title/text()').extract():
           #print h3 + "***********" + ' &lt;' + response.url + '&gt;'
           sliced_url = response.url.split('/')[2]
  
  
      for url in sel.xpath('//a/@href').extract():
  
          if (url.startswith('/')):
  
              url = 'http://' + sliced_url + url
  
  
  
          if (url in parsed_urls or len(url) &gt; 250):
              continue
  
          parsed_urls.append(url)
  
          if tld in url:
              domain_urls.append(url)
              yield Request(url, callback=self.parse)
  
          #print parsed_urls
          for keyword in Keyword_Dictionary:
  
              if (url.startswith('http') and (tld in url)):
                  if (self.Search_keyword_in_url(keyword, response)):
                      if keyword not in Url_Dictionary:
                          Url_Dictionary[keyword] = []
                      Url_Dictionary[keyword].append(url)
                      #print keyword + " " + "Detected"
  
  
  
          for keyword in Url_Dictionary:
  
              Attribute_Key = []
              Attribute_Key = Keyword_Dictionary.get(keyword)
              Attribute_Key_Value = Url_Dictionary.get(keyword)
              for key in Attribute_Key:
                  if key not in Attribute_Dictionary:
                      Attribute_Dictionary[key] = []
                      print key
                      print "\n"
                      for value in Attribute_Key_Value:
                          if value not in Attribute_Dictionary.get(key):
                              Attribute_Dictionary[key].append(value)
                              print key + " " + "Just Appended"
                              item = Website()
                              Modified_Key = key.replace(" ","_")
                              item[Modified_Key] = response.url
                              print item[Modified_Key]
  
          print Attribute_Dictionary
          # Json Code
          fileptr = open('keywords_spider.json','a')
          json.dump(Attribute_Dictionary, fileptr, indent=4)
          print "Created keywords_spiders.json.."
          fileptr.close()
  
  
  
  
  def Search_keyword_in_url(self, keyword, response):
  
      sel = Selector(response)
  
      text_list = sel.xpath('//div/p/text()').extract()
      for text in text_list:
          if text.find(keyword) &gt; -1:
              return True
      return False
  </code></pre>