title: showing kwic in scrapy item
body: <p>Good afternoon all,</p>
  
  <p>I am trying to write a spider that will give me results with keywords in context. I've managed to find this link: <a href="https://stackoverflow.com/questions/4319473/python-truncate-text-around-keyword">python truncate text around keyword</a>
  which explains a way to do this, I've moulded it a little to suite my needs but am now no longer getting any scraped data in my database. (the only thing that's changed is I've added the code to grab the kwic as one of my Items).</p>
  
  <p>My Spider:</p>
  
  <pre><code># tabbing in python is apparently VERY important so be aware and make sure 
  # things that should line up do so
  
  # import the CrawlSpider Class, along with it's Rules, (this lets us recursively
  # crawl pages)
  
  from scrapy.contrib.spiders import CrawlSpider, Rule
  
  #import the link extractor, this extracts links from pages
  
  from scrapy.contrib.linkextractors import LinkExtractor
  
  # import our items as defined in items.py
  
  from basic.items import BasicItem
  
  # import re which allows us to compare strings
  
  import re
  
  # create a new Spider with the CrawlSpider Class
  
  class BasicSpiderSpider(CrawlSpider):
  
      # Name of the spider, this is used to run it, (i.e Scrapy Crawl basic_spider)
  
      name = "basic_spider"
  
      # domains that the spider is allowed to crawl over
  
      allowed_domains = ["news24.com"]
  
      # where to start crawling from
  
      start_urls = [
          'http://www.news24.com/SouthAfrica/News/Six-things-to-know-about-the-illegal-mining-boom-20140626',
      ]
  
      # Rules for the link extractor, (i.e where it's allowed to look for links, 
      # what to do once it's found them, and whether it's allowed to follow them
  
      rules = (Rule (LinkExtractor(), callback="parse_items", follow= True),
      )
  
     # defining the callback function
  
      def parse_items(self, response):
  
          # defines the Top level XPath where all of our information can be found, needs to be
          # as specific as possible to avoid duplicates
  
          for title in response.xpath('//*[@id="aspnetForm"]'):
  
              # List of keywords to search through.
  
              key = re.compile("joburg|durban|children", re.IGNORECASE)
  
              # extracting the data to compare with the keywords, this is for the 
              # headlines, the join converts it from a list type to a string type
  
              headlist = title.xpath('//*[@id="article_special"]//h1/text()').extract()
              head = ''.join(headlist)
  
              # and this is for the article.
  
              artlist = title.xpath('//*[@id="article-body"]//text()').extract()
              art = ''.join(artlist)
  
              # if any keywords are found in the headline:
  
              if key.search(head):
  
                      # define the top level xpath again as python won't look outside 
                      # it's current fuction
  
                      for thing in response.xpath('//*[@id="aspnetForm"]'):
  
      # using this because i'm wanting to show the context of returned articles, found this here:
      # https://stackoverflow.com/questions/4319473/python-truncate-text-around-keyword
  
                          def find_with_context(haystack, needle, context_length, escape=True):
                              if escape:
                                  needle = re.escape(needle)
                              return re.findall(r'\b(.{,%d})\b(%s)\b(.{,%d})\b' % (context_length, needle, context_length), haystack)
  
  
                          # fills the items defined in items.py with relevant data
  
                          item = BasicItem()
                          item['Headline'] = thing.xpath('//*[@id="article_special"]//h1/text()').extract()
  
                          #calls find_with_context
  
                          item["Article"] = find_with_context("joburg|durban|children", (art), 50)
      #                    item["Article"] = thing.xpath('//*[@id="article-body"]//text()').extract
                          item["Date"] = thing.xpath('//*[@id="spnDate"]/text()').extract()
                          item["Link"] = response.url
  
                          # I found that even with being careful about my XPaths I
                          # still got empty fields and lines, the below fixes that
  
                          if item['Headline']:
                              if item["Article"]:
                                  if item["Date"]:
                                      yield item
  
              # if the headline item doesn't match, check the article item.
  
              elif key.search(art):
                  #if last_crawled &gt; response.xpath('//*[@id="spnDate"]/text()').extract():
                      for thing in response.xpath('//*[@id="aspnetForm"]'):
  
      # using this because i'm wanting to show the context of returned articles, found this here:
      # https://stackoverflow.com/questions/4319473/python-truncate-text-around-keyword                        
  
                          def find_with_context(haystack, needle, context_length, escape=True):
                              if escape:
                                  needle = re.escape(needle)
                              return re.findall(r'\b(.{,%d})\b(%s)\b(.{,%d})\b' % (context_length, needle, context_length), haystack)
  
  
                          item = BasicItem()
                          item['Headline'] = thing.xpath('//*[@id="article_special"]//h1/text()').extract()
                          #item["Article"] = thing.xpath('//*[@id="article-body"]/p[1]/text()').extract()
                          item["Article"] = find_with_context("joburg|durban|children", (art), 50)
                          item["Date"] = thing.xpath('//*[@id="spnDate"]/text()').extract()
                          item["Link"] = response.url
  
                          if item['Headline']:
                              if item["Article"]:
                                  if item["Date"]:
                                      yield item
  </code></pre>
  
  <p>My Log File can be downloaded here: <a href="http://1drv.ms/1y0umXn" rel="nofollow noreferrer">http://1drv.ms/1y0umXn</a></p>
  
  <p>It looks as if it's no longer recognising the page: <a href="http://www.news24.com/SouthAfrica/News/Gunshots-screaming-in-Manenberg-overnight-20150407" rel="nofollow noreferrer">http://www.news24.com/SouthAfrica/News/Gunshots-screaming-in-Manenberg-overnight-20150407</a></p>
  
  <p>which does match the search criteria.</p>
  
  <p>any help would be greatly appreciated</p>
  
  <p>Kind regards,
  Grant</p>