title: How to evaluate if an extracted link is a subpath
body: <p>I'm using scrapy to crawl some pages. I am using python 2.7. 
  The spider returns the response object, and I am inspecting URLs found on the page. I want to restrict the spider to only follow URLS that are subpaths of a location I specify.</p>
  
  <p>For instance, I would like to specify that the spider should only follows links below www.google.com/policies/privacy/ </p>
  
  <p>The links extracted from the response object follow a number of different conventions. </p>
  
  <p>E.g. </p>
  
  <ul>
  <li>../../policies/privacy/</li>
  <li>../../policies/privacy/example/collect-information.html</li>
  <li>#infocollect</li>
  <li>/intl/en_uk/policies/privacy/google_privacy_policy_en_uk.pdf</li>
  <li>//myaccount.google.com/</li>
  <li><a href="https://support.google.com/policies/troubleshooter/2990837?hl=en-GB" rel="nofollow">https://support.google.com/policies/troubleshooter/2990837?hl=en-GB</a></li>
  </ul>
  
  <p>I can't work out how to do it. I have just used a simple find method on the string. It's not robust or that clever it seems to me. </p>
  
  <pre><code>import scrapy
  
  class googleSpider(scrapy.Spider):
      name = "google"
      allowed_domains = ["google.co.uk"]
      start_urls = [
          "http://www.google.co.uk/intl/en/policies/privacy/"
      ]
  
      def parse(self, response):
          for href in response.xpath('//a/@href').extract():
              if href.find('/policies/privacy/') != -1:
                  yield scrapy.Request(response.urljoin(href), callback=self.parse_dir_contents)
  
      def parse_dir_contents(self, response):
          pass
  </code></pre>