title: How does Linkedin make difference between user requests and crawler requests?
body: <p>When I try to <strong>download one page from Linkedin</strong> with the following command:</p>
  
  <pre><code>curl -I https://www.linkedin.com/company/google
  </code></pre>
  
  <p>I get a <strong>999 status code</strong>:</p>
  
  <pre><code>HTTP/1.1 200 Connection established
  
  HTTP/1.1 999 Request denied
  Date: Tue, 30 Aug 2016 08:19:35 GMT
  X-Li-Pop: prod-tln1-hybla
  Content-Length: 1629
  Content-Type: text/html
  </code></pre>
  
  <p>Since users using a browser can access to Linkedin pages, it means that <strong>they can make difference between robots and users</strong>. </p>
  
  <p>Else users would not be allow to access Linkedin pages due to the following lines at the end of <a href="https://www.linkedin.com/robots.txt" rel="nofollow">robots.txt</a>:</p>
  
  <pre><code>User-agent: *
  Disallow: /
  </code></pre>
  
  <p>So, Linkedin can make difference between requests coming from browsers and others. <strong><em>How do they do that ?</em></strong></p>