title: R: Check existence of url, problems with httr:GET() and url.exists()
body: <p>I have a list of about 13,000 URLs that I want to extract info from, however, not every URL actually exists. In fact the majority don't. I have just tried passing all 13,000 urls through <code>html()</code> but it takes a long time. I am trying to work out how to see if the urls actually exist before parsing them to <code>html()</code>. I have tried using <code>httr</code> and <code>GET()</code> functions, as well as <code>rcurls</code> and <code>url.exists()</code> functions. For some reason <code>url.exist()</code> always returns <code>FALSE</code> values even when the URL does exist, and the way I am using <code>GET()</code> always returns a success, I think this is because the page is being redirected.</p>
  
  <p>The following URLs represent the type of pages I am parsing, the first does not exist </p>
  
  <pre><code>urls &lt;- data.frame('site' = 1:3, 'urls' = c('https://www.deakin.edu.au/current-students/unitguides/UnitGuide.php?year=2015&amp;semester=TRI-1&amp;unit=SLE010', 
                              'https://www.deakin.edu.au/current-students/unitguides/UnitGuide.php?year=2015&amp;semester=TRI-2&amp;unit=HMM202',
                              'https://www.deakin.edu.au/current-students/unitguides/UnitGuide.php?year=2015&amp;semester=TRI-2&amp;unit=SLE339'))
  
  urls$urls &lt;- as.character(urls$urls)
  </code></pre>
  
  <p>For <code>GET()</code>, the problem is that the second URL doesn't actually exist but it is redirected and therefore returns a "success".</p>
  
  <pre><code> urls$urlExists &lt;- sapply(1:length(urls[,1]), 
                       function(x) ifelse(http_status(GET(urls[x, 'urls']))[[1]] == "success", 1, 0))
  </code></pre>
  
  <p>For <code>url.exists()</code>, I get three FALSE returned even though the first and third urls do exist.</p>
  
  <pre><code> urls$urlExists2 &lt;- sapply(1:length(urls[,1]), function(x) url.exists(urls[x, 'urls']))
  </code></pre>
  
  <p>I checked these two posts <a href="https://stackoverflow.com/questions/29130237/rcurl-url-exists-returns-false-when-url-does-exists">1</a>, <a href="https://stackoverflow.com/questions/23139357/how-to-determine-if-a-url-object-in-r-base-package-returns-404-not-found">2</a> but I would prefer not to use a useragent simply because I am not sure how to find mine or whether it would change for different people using this code on other computers. Therefore making the code harder to pick up and use by others. Both posts answers suggest using <code>GET()</code> in <code>httr</code>. It seems that <code>GET()</code> is probably the preferred method but I would need to figure out how to deal with the redirection issue. </p>
  
  <p>Can anyone suggest a good way in R to test the existence of a URL before parsing them to <code>html()</code>? I would also be happy for any other suggested work around for this issue.</p>
  
  <p>UPDATE:</p>
  
  <p>After looking into the returned value from <code>GET()</code> I figured out a work around, see answers for details.</p>