title: Scrapy: deny/ignore link with specific class attribute
body: <p>I am crawling a large website with scrapy and I am looking for a way to deny all  tags with an attribute class="AdvSearchKeyword_clearall".</p>
  
  <p>If it is not possible to deny an <code>&lt;a&gt;</code> tag with class="AdvSearchKeyword_clearall", would it be possible to parse the "AdvSearchKeyword_clearall" to filter out all links with the specific attribute later?</p>
  
  <hr>
  
  <pre><code>from scrapy.selector import HtmlXPathSelector
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  
  from wallspider.items import Website
  
  
  class ComSpider(CrawlSpider):
      name = "browsepages"
      allowed_domains = ["www.mydomain.com"]
      start_urls = ["http://www.mydomain.com",]
  
      rules = (
      Rule (SgmlLinkExtractor(allow=('/browse/', ),)
      , callback="parse_items", follow= True),
      Rule(SgmlLinkExtractor(allow=(),unique=True,deny=('/[1-9]$', '(bti=)[1-9]+(?:\.[1-9]*)?', '(sort_by=)[a-zA-Z]', '(sort_by=)[1-9]+(?:\.[1-9]*)?', '(ic=32_)[1-9]+(?:\.[1-9]*)?', '(ic=60_)[0-9]+(?:\.[0-9]*)?', '(search_sort=)[1-9]+(?:\.[1-9]*)?', 'browse-ng.do\?', '/page/', '/ip/', 'out\+value', 'fn=', 'customer_rating', 'special_offers', 'search_sort=&amp;', 'facet=' ))),
      )
  
      def parse_items(self, response):
          hxs = HtmlXPathSelector(response)
          sites = hxs.select('//html')
          items = []
  
          for site in sites:
              item = Website()
              item['url'] = response.url
              item['referer'] = response.request.headers.get('Referer')
              item['canonical'] = site.xpath('//head/link[@rel="canonical"]/@href').extract()
              item['robots'] = site.select('//meta[@name="robots"]/@content').extract()
              items.append(item)
  
          return items
  </code></pre>