title: Scrapy - Does urlparse.urljoin behave in the same way as str.join?
body: <p>I am trying to use <code>urlparse.urljoin</code> within a Scrapy spider to compile a list of urls to scrape. Currently, my spider is returning nothing, but not throwing any errors. So I am trying to check that I am compiling the urls corectly. </p>
  
  <p>My attempt was to test this in idle using <code>str.join</code>, as below:</p>
  
  <pre><code>&gt;&gt;&gt; href = ['lphs.asp?id=598&amp;city=london',
   'lphs.asp?id=480&amp;city=london',
   'lphs.asp?id=1808&amp;city=london',
   'lphs.asp?id=1662&amp;city=london',
   'lphs.asp?id=502&amp;city=london',]
  &gt;&gt;&gt; for x in href:
      base = "http:/www.url-base.com/destination/"
      final_url = str.join(base, x)
      print(final_url)
  </code></pre>
  
  <p>A one line of what that returns:</p>
  
  <p><code>lhttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/hhttp:/www.url-base.com/destination/shttp:/www.url-base.com/destination/.http:/www.url-base.com/destination/ahttp:/www.url-base.com/destination/shttp:/www.url-base.com/destination/phttp:/www.url-base.com/destination/?http:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/5http:/www.url-base.com/destination/9http:/www.url-base.com/destination/8http:/www.url-base.com/destination/&amp;http:/www.url-base.com/destination/chttp:/www.url-base.com/destination/ihttp:/www.url-base.com/destination/thttp:/www.url-base.com/destination/yhttp:/www.url-base.com/destination/=http:/www.url-base.com/destination/lhttp:/www.url-base.com/destination/ohttp:/www.url-base.com/destination/nhttp:/www.url-base.com/destination/dhttp:/www.url-base.com/destination/ohttp:/www.url-base.com/destination/n</code></p>
  
  <p>I think that from my example it is obvious that <code>str.join</code> does not behave in the same way - if it does then there this is why my spider is not following these links! - however, it would be good to have confirmation on that.</p>
  
  <p>If this is not the right way to test, how can I test this process? </p>
  
  <p><strong>Update</strong> 
  Attempt using <code>urlparse.urljoin</code> below: 
  from urllib.parse import urlparse</p>
  
  <pre><code>    &gt;&gt;&gt; from urllib.parse import urlparse
      &gt;&gt;&gt; for x in href:
          base = "http:/www.url-base.com/destination/"
          final_url = urlparse.urljoin(base, x)
          print(final_url)
  </code></pre>
  
  <p>Which is throwing <code>AttributeError: 'function' object has no attribute 'urljoin'</code></p>
  
  <p><strong>Update - the spider function in question</strong></p>
  
  <pre><code>def parse_links(self, response): 
      room_links = response.xpath('//form/table/tr/td/table//a[div]/@href').extract() # insert xpath which contains the href for the rooms 
      for link in room_links:
          base_url = "http://www.example.com/followthrough"
          final_url = urlparse.urljoin(base_url, link)
          print(final_url)
          # This is not joing the final_url right
          yield Request(final_url, callback=parse_links)
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>I just tested again in idle:</p>
  
  <pre><code>&gt;&gt;&gt; from urllib.parse import urljoin
  &gt;&gt;&gt; from urllib import parse
  &gt;&gt;&gt; room_links = ['lphs.asp?id=562&amp;city=london',
   'lphs.asp?id=1706&amp;city=london',
   'lphs.asp?id=1826&amp;city=london',
   'lphs.asp?id=541&amp;city=london',
   'lphs.asp?id=1672&amp;city=london',
   'lphs.asp?id=509&amp;city=london',
   'lphs.asp?id=428&amp;city=london',
   'lphs.asp?id=614&amp;city=london',
   'lphs.asp?id=336&amp;city=london',
   'lphs.asp?id=412&amp;city=london',
   'lphs.asp?id=611&amp;city=london',]
  &gt;&gt;&gt; for link in room_links:
      base_url = "http:/www.url-base.com/destination/"
      final_url = urlparse.urljoin(base_url, link)
      print(final_url)
  </code></pre>
  
  <p>Which threw this: </p>
  
  <pre><code>Traceback (most recent call last):
    File "&lt;pyshell#34&gt;", line 3, in &lt;module&gt;
      final_url = urlparse.urljoin(base_url, link)
  AttributeError: 'function' object has no attribute 'urljoin'
  </code></pre>