title: Scrapy LinkExtractor crawling links that use parent directory
body: <p>Using a basic <a href="https://docs.scrapy.org/en/latest/topics/spiders.html#crawlspider" rel="nofollow noreferrer">CrawlerSpider</a> in Scrapy, I'm trying to crawl a page. The related links in the page that I want to crawl all start with the parent directory notation <code>..</code> instead of the full domain. </p>
  
  <p>For example, if I'm starting with page <code>https://www.mytarget.com/posts/4/friendly-url</code>, and I want to crawl every post in <code>/posts</code>, the related links on that page will be: </p>
  
  <pre><code>'../55/post-name'
  '../563/another-name'
  </code></pre>
  
  <p>instead of:</p>
  
  <pre><code>'posts/55/post-name'
  'posts/563/another-name'
  </code></pre>
  
  <p>or what would be better:</p>
  
  <pre><code>'https://www.mytarget.com/posts/55/post-name'
  'https://www.mytarget.com/posts/563/another-name'
  </code></pre>
  
  <p>Removing <code>mytarget.com</code> from <code>allowed_domains</code> doesn't seem to help. The crawler won't find new links on the website that match the <code>..</code> parent directory link reference.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import scrapy
  from scrapy.linkextractors import LinkExtractor
  from scrapy.spiders import CrawlSpider, Rule
  from exercise_data_collector.items import Post
  
  class MyCrawlerSpider(CrawlSpider):
      name = 'my_crawler'
      allowed_domains = ['mytarget.com']
      start_urls = ['https://www.mytarget.com/posts/4/friendly-url']
  
      rules = (
          Rule(LinkExtractor(allow=r'posts/[0-9]+/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
          Rule(LinkExtractor(allow=r'/posts\/[0-9]+\/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
          Rule(LinkExtractor(allow=r'/..\/[0-9]+\/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
      )
  
      def parse(self, response):
          links = self.le1.extract_links(response)
  
          item = Post()
          item["page_title"] = response.xpath('//title/text()').get()
          item["name"] = response.xpath("//div[@class='container']/div[@class='row']/div[1]/div[1]/text()[2]").get().replace('-&gt;','').strip()
          item['difficulty'] = response.xpath("//p[strong[contains(text(), 'Difficulty')]]/text()").get().strip()
  
          return item
  </code></pre>
  
  <p>I'm not sure exactly how to configure the regex to get the related links or to even test if the regex is working outside of <code>regexr.com</code>.</p>
  
  <p>How can I crawl pages like this?</p>