title: DNS lookup failed: address 'your.proxy.com' not found: [Errno -5] No address associated with hostname
body: <p>This question is an extension of the resolved question here, ie. Crawling linkedin while authenticating with scrapy. <a href="https://stackoverflow.com/questions/10953991/crawling-linkedin-while-authenticated-with-scrapy">Crawling LinkedIn while authenticated with Scrapy</a> @Gates</p>
  
  <p>While I keep the base of the script the same, only adding my own session_key and session_password - and after changing the start url particular to my use-case, as below.</p>
  
  <pre><code>class LinkedPySpider(InitSpider):
      name = 'Linkedin'
      allowed_domains = ['linkedin.com']
      login_page = 'https://www.linkedin.com/uas/login'
      start_urls=["http://www.linkedin.com/nhome/"]
  
  [Also tried with this start url] 
  start_urls =
  ["http://www.linkedin.com/profile/view?id=38210724&amp;trk=nav_responsive_tab_profile"]
  </code></pre>
  
  <p>I also tried changing the start_url to the second one(commented), to see if I could start scraping from my own profile page, I was unable to do so. </p>
  
  <pre><code>**Error that I get** - 
  scrapy crawl Linkedin
  **2013-07-29 11:37:10+0530 [Linkedin] DEBUG: Retrying &lt;GET http://www.linkedin.com/nhome/&gt; (failed 1 times): DNS lookup failed: address 'your.proxy.com' not found: [Errno -5] No address associated with hostname.**
  
  
  **To see if the Name space was resolved, I tried -:**
  nslookup www.linkedin.com #works
  nslookup www.linkedin.com/uas/login # I think the depth of pages within a main website, does not resolve, and is normal right ?
  
  Then I also tried to see if the error could have been due to Name Server not resolving and appended the Nameservers as below .
  echo $http_proxy #gives http://username:password@your.proxy.com:80
  sudo vi /etc/resolv.conf
  and appended the free fast dns nameservers IP address as follows to this file :
  nameserver 208.67.222.222
  nameserver 208.67.220.220
  nameserver 202.51.5.52
  </code></pre>
  
  <p>I am not too good with NS conflicts and DNS lookup failures, but could this be due to the fact that I am in a VM - though other scraping projects seemed to work just fine ? </p>
  
  <p>My base use-case is to be able to extract connections and the list of companies they worked at, and a bunch of other attributes. So, I want to crawl/paginate from the "Connections" (All) in the main profile page, which does NOT show up if I use public profile in the start_url, ie. scrapy shell <a href="http://www.linkedin.com/in/ektagrover" rel="nofollow noreferrer">http://www.linkedin.com/in/ektagrover</a>
  On passing legitimate XPath via hxs.select - this seems to work, but NOT if I used it along with a spider, since it did not meet my base-usecase(As below)</p>
  
  <p><strong>Question :</strong> Is there something wrong with my start_url, or is it just the way that I am "assuming that post the authentication the start_page could come to potentially <strong>ANY</strong> webpage in that site, when I redirect it post authentication at "<a href="https://www.linkedin.com/uas/login" rel="nofollow noreferrer">https://www.linkedin.com/uas/login</a>"</p>
  
  <p><strong>Work-environment</strong> - I am on Oracle VM Virtual Box with ubuntu 12.04 LTS with Python 2.7.3, with Scrapy 0.14.4</p>
  
  <p><strong>What worked/ Answer --</strong> Looks like my proxy server was incorrectly pointing to 
  echo $http_proxy - which gives <a href="http://username:password@your.proxy.com:80" rel="nofollow noreferrer">http://username:password@your.proxy.com:80</a>
  [Unset the environment variable $http_proxy ] Just did " http_proxy= " , which unsets the proxy then did echo $http_proxy , which gives null to confirm . Post that just did scrapy crawl Linkedin, which worked through the authentication module. Though I am getting stuck here and there on selenium, but that's for another question. Thank you, @warwaruk</p>