title: How To Rotate Proxies and IP Addresses using R and rvest
body: <p>I'm doing some scraping, but as I'm parsing approximately 4000 URL's, the website eventually detects my IP and blocks me every 20 iterations.</p>
  <p>I've written a bunch of <code>Sys.sleep(5)</code> and a <code>tryCatch</code> so I'm not blocked too soon.</p>
  <p>I use a VPN but I have to manually disconnect and reconnect it every now and then to change my IP. That's not a suitable solution with such a scraper supposed to run all night long.</p>
  <p>I think rotating a proxy should do the job.</p>
  <p>Here's my current code (a part of it at least) :</p>
  <pre><code>library(rvest)
  library(dplyr)
  
  scraped_data = data.frame()
  
  for (i in urlsuffixes$suffix)
    {
    
    tryCatch({
      message(&quot;Let's scrape that, Buddy !&quot;)
      
      Sys.sleep(5)
   
      doctolib_url = paste0(&quot;https://www.website.com/test/&quot;, i)
  
      page = read_html(site_url)
      
      links = page %&gt;%
        html_nodes(&quot;.seo-directory-doctor-link&quot;) %&gt;%
        html_attr(&quot;href&quot;)
      
      Sys.sleep(5)
      
      name = page %&gt;%
        html_nodes(&quot;.seo-directory-doctor-link&quot;) %&gt;%
        html_text()
      
      Sys.sleep(5)
      
      job_title = page %&gt;%
        html_nodes(&quot;.seo-directory-doctor-speciality&quot;) %&gt;%
        html_text()
      
      Sys.sleep(5)
      
      address = page %&gt;%
        html_nodes(&quot;.seo-directory-doctor-address&quot;) %&gt;%
        html_text()
      
      Sys.sleep(5)
      
      scraped_data = rbind(scraped_data, data.frame(links,
                                                    name,
                                                    address,
                                                    job_title,
                                                    stringsAsFactors = FALSE))
      
    }, error=function(e){cat(&quot;Houston, we have a problem !&quot;,&quot;\n&quot;,conditionMessage(e),&quot;\n&quot;)})
    print(paste(&quot;Page : &quot;, i))
  }
  </code></pre>