title: scrapy: Remove some elements from an xpath selector
body: <p>I'm using Scrapy to crawl a site with some odd formatting conventions. The basic idea is that I want all the text and sub-elements of a certain div, EXCEPT a few div in the middle. Here is the piece of code below :-</p>
  
  <pre><code>&lt;div align="center" class="article"&gt;&lt;!--wanted--&gt;
      &lt;img src="http://i.imgur.com/12345.jpg" width="500" alt="abcde" title="abcde"&gt;&lt;br&gt;&lt;br&gt;     
      &lt;div style="text-align:justify"&gt;&lt;!--wanted--&gt;
          Sample Text&lt;br&gt;&lt;br&gt;Demo: &lt;a href="http://www.example.com/?http://example.com/item/asash/asdas-asfasf-afaf.html" target="_blank"&gt;http://example.com/dfa/asfa/aasfa&lt;/a&gt;&lt;br&gt;&lt;br&gt;
          &lt;div class="quote"&gt;&lt;!--wanted--&gt;
              http://www.coolfiles.ro/download/kleo13.rar/1098750&lt;br&gt;http://www.ainecreator.com/files/0MKOGM6D/kleo13.rar_links&lt;br&gt;
          &lt;/div&gt;
          &lt;br&gt;
          &lt;div align="left"&gt;&lt;!--not wanted--&gt;
              &lt;div id="ratig-layer-2249"&gt;&lt;!--not wanted--&gt;
                  &lt;div class="rating"&gt;&lt;!--not wanted--&gt;
                      &lt;ul class="unit-rating"&gt;
                          &lt;li class="current-rating" style="width:80%;"&gt;80&lt;/li&gt;
                          &lt;li&gt;&lt;a href="#" title="Bad" class="r1-unit" onclick="doRate('1', '2249'); return false;"&gt;1&lt;/a&gt;&lt;/li&gt;
                          &lt;li&gt;&lt;a href="#" title="Poor" class="r2-unit" onclick="doRate('2', '2249'); return false;"&gt;2&lt;/a&gt;&lt;/li&gt;
                          &lt;li&gt;&lt;a href="#" title="Fair" class="r3-unit" onclick="doRate('3', '2249'); return false;"&gt;3&lt;/a&gt;&lt;/li&gt;
                          &lt;li&gt;&lt;a href="#" title="Good" class="r4-unit" onclick="doRate('4', '2249'); return false;"&gt;4&lt;/a&gt;&lt;/li&gt;
                          &lt;li&gt;&lt;a href="#" title="Excellent" class="r5-unit" onclick="doRate('5', '2249'); return false;"&gt;5&lt;/a&gt;&lt;/li&gt;
                      &lt;/ul&gt;
                  &lt;/div&gt;
                  (votes: &lt;span id="vote-num-id-2249"&gt;3&lt;/span&gt;)
              &lt;/div&gt;
          &lt;/div&gt;
          &lt;div class="reln"&gt;&lt;!--not wanted--&gt;
              &lt;strong&gt;
                  &lt;h4&gt;Related News:&lt;/h4&gt;
              &lt;/strong&gt;
              &lt;li&gt;&lt;a href="http://www.example.com/themes/tf/a-b-c-d.html"&gt;1&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href="http://www.example.com/plugins/codecanyon/a-b-c-d"&gt;2&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href="http://www.example.com/themes/tf/a-b-c-d.html"&gt;3&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href="http://www.example.com/plugins/codecanyon/a-b-c-d.html"&gt;4&lt;/a&gt;&lt;/li&gt;
              &lt;li&gt;&lt;a href="http://www.example.com/plugins/codecanyon/a-b-c-d.html"&gt;5&lt;/a&gt;&lt;/li&gt;
          &lt;/div&gt;
      &lt;/div&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>The final output should look like :-</p>
  
  <pre><code>&lt;div align="center" class="article"&gt;&lt;!--wanted--&gt;
      &lt;img src="http://i.imgur.com/12345.jpg" width="500" alt="abcde" title="abcde"&gt;&lt;br&gt;&lt;br&gt;     
      &lt;div style="text-align:justify"&gt;&lt;!--wanted--&gt;
          Sample Text&lt;br&gt;&lt;br&gt;Demo: &lt;a href="http://www.example.com/?http://example.com/item/asash/asdas-asfasf-afaf.html" target="_blank"&gt;http://example.com/dfa/asfa/aasfa&lt;/a&gt;&lt;br&gt;&lt;br&gt;
          &lt;div class="quote"&gt;&lt;!--wanted--&gt;
              http://www.coolfiles.ro/download/kleo13.rar/1098750&lt;br&gt;http://www.ainecreator.com/files/0MKOGM6D/kleo13.rar_links&lt;br&gt;
          &lt;/div&gt;
          &lt;br&gt;
      &lt;/div&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>Here is the piece of my Scrapy code. Please suggest the addition to this script :-</p>
  
  <pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  from scrapy.selector import HtmlXPathSelector
  from isbullshit.items import IsBullshitItem
  
  
  class IsBullshitSpider(CrawlSpider):
      """ General configuration of the Crawl Spider """
      name = 'isbullshitwp'
      start_urls = ['http://example.com/themes'] # urls from which the spider will start crawling
      rules = [Rule(SgmlLinkExtractor(allow=[r'page/\d+']), follow=True), 
          # r'page/\d+' : regular expression for http://example.com/page/X URLs
          Rule(SgmlLinkExtractor(allow=[r'\w+']), callback='parse_blogpost')]
          # r'\d{4}/\d{2}/\w+' : regular expression for http://example.com/YYYY/MM/title URLs
  
      def parse_blogpost(self, response):
          hxs = HtmlXPathSelector(response)
          item = IsBullshitItem()
          item['title'] = hxs.select('//span[@class="storytitle"]/text()').extract()[0]
          item['article_html'] = hxs.select("//div[@class='article']").extract()[0]
  
          return item
  </code></pre>
  
  <p>Here are the following xpath that I experimented with but did not get the desired results :-</p>
  
  <pre><code>item['article_html'] = hxs.select("//div[@class='article']").extract()[0]
  item['article_html'] = hxs.select("//div[@class='article']/following::node() [not(preceding::div[@class='reln']) and not(@class='reln')]").extract()[0]
  item['article_html'] = hxs.select("//div[@class='article']/div[@class='reln']/preceding-sibling::node()[preceding-sibling::div[@class='quote']]").extract()[0]
  item['article_html'] = hxs.select("//div[@class='article']/following::node() [not(preceding::div[@class='reln'])]").extract()[0]
  item['article_html'] = hxs.select("//div[@class='article']/div[@class='quote']/*[not(self::div[@class='reln'])]").extract()[0]
  item['article_html'] = hxs.select("//div[@class='article']/*[(self::name()='reln'])]").extract()[0]
  </code></pre>
  
  <p>Thanks in advance...</p>