scrapy and relative paths
Question: I am not straightforward with python, I have been trying for hours now to cut
off few char from multiple elements in a list. All scrapped links that I want
to follow with my spider are relative, here are just few lines of my output:

    
    
     [u'../../../info/Auto/Dutch/']
     [u'../../../info/Automazing/All_Star/']
     [u'../../../info/AutoW/Passion/'] 
    

I am trying to get rid of

    
    
     "../../.."
    

This is my code:

    
    
      def parse(self, response):
        hxs = HtmlXPathSelector(response)
        links = hxs.select('//div[@class="partInnerDiv"]/table/tbody/tr')
        items = []
        for link in links:
           item = myItem()
           item ['url'] = link.select('th/a/@href').extract()   
           print complete_url(item ['url'])
           items.append(item)            
        return items
    
      def complete_url(string):
        url = string
       #if string.startswith("../"):
       #   return string[5:]
        return url.replace('../../', '')
    

From the code above you can see that I tried few methods, but nothing works
for me :(

    
    
       exceptions.AttributeError: 'list' object has no attribute 'replace'
    

Any advice appreciated.


Answer: You are looking for
[`urlparse.urljoin()`](http://docs.python.org/2/library/urlparse.html#urlparse.urljoin):

    
    
    import urlparse
    
    urlparse.urljoin(u'http://www.mysite.com/a/b/c/', u'../../../info/Auto/Dutch/')
    
    # => u'http://www.mysite.com/info/Auto/Dutch/'
    

In your callback it would become:

    
    
      def parse(self, response):
        hxs = HtmlXPathSelector(response)
        links = hxs.select('//div[@class="partInnerDiv"]/table/tbody/tr')
        items = []
        for link in links:
           item = myItem()
           item ['url'] = [urlparse.urljoin(response.url, u)
                           for u in link.select('th/a/@href').extract()]
           items.append(item)            
        return items
    

This code assumes Python 2.x; in Python 3, the function has been renamed to
`urllib.parse.urljoin()`, but Scrapy does not currently work with Python 3.

