title: “Error in open.connection(x, "rb") : Timeout was reached”
body: <p>When I do some webscraping (using a for loop to scrap multiple pages), sometimes, after scraping the 35th out of 40 pages, I have the following error:</p>
  
  <blockquote>
    <p>“Error in open.connection(x, "rb") : Timeout was reached”</p>
  </blockquote>
  
  <p>And sometimes I receive in addition this message:</p>
  
  <blockquote>
    <p>“In addition: Warning message: closing unused connection 3”</p>
  </blockquote>
  
  <p>Below a list of things I would like to clarify:</p>
  
  <p>1) I have read it might need to define explicitly the user agent. I have tried that with:</p>
  
  <pre><code>read_html(curl('www.link.com', handle = curl::new_handle("useragent" = "Mozilla/5.0")))
  </code></pre>
  
  <p>but it did not change anything.</p>
  
  <p>2) I noticed that when I turn on a VPN, and change location, sometimes my scraping works without any error. I would like to understand why?</p>
  
  <p>3) I have also read it might depend of the proxy. How would like to understand how and why?</p>
  
  <p>4) In addition to the error I have, I would like to understand this warning, has it might be a clue that leads to understand the error:</p>
  
  <blockquote>
    <p>Warning message: closing unused connection 3</p>
  </blockquote>
  
  <p>Does that mean that when I am doing webscraping I should somehow at the end call a function to close a connection?</p>
  
  <p>I have already read the following posts on stackoverflow but there is no clear resolution:</p>
  
  <p><a href="https://stackoverflow.com/questions/39056103/iterating-rvest-scrape-function-gives-error-in-open-connectionx-rb-time">Iterating rvest scrape function gives: &quot;Error in open.connection(x, &quot;rb&quot;) : Timeout was reached&quot;</a></p>
  
  <p><a href="https://stackoverflow.com/questions/33295686/rvest-error-in-open-connectionx-rb-timeout-was-reached">rvest Error in open.connection(x, &quot;rb&quot;) : Timeout was reached</a></p>
  
  <p><a href="https://stackoverflow.com/questions/37794368/error-in-open-connectionx-rb-couldnt-connect-to-server">Error in open.connection(x, &quot;rb&quot;) : Couldn&#39;t connect to server</a></p>