title: how to download a large binary file with RCurl *after* server authentication
body: <p>i originally asked <a href="https://stackoverflow.com/questions/17306695/httr-get-function-running-out-of-space-when-downloading-a-large-file">this question</a> about performing this task with the <code>httr</code> package, but i don't think it's possible using <code>httr</code>.  so i've re-written my code to use <code>RCurl</code> instead -- but i'm still tripping up on something probably related to the <code>writefunction</code>.. but i really don't understand why.</p>
  
  <p>you should be able to reproduce my work by using the 32-bit version of R, so you hit memory limits if you read anything into RAM.  i need a solution that downloads directly to the hard disk.</p>
  
  <p>to start, this code to <em>works</em> -- the zipped file is appropriately saved to the disk.</p>
  
  <pre><code>library(RCurl)
  filename &lt;- tempfile()
  f &lt;- CFILE(filename, "wb")
  url &lt;- "http://www2.census.gov/acs2011_5yr/pums/csv_pus.zip"
  curlPerform(url = url, writedata = f@ref)
  close(f)
  # 2.1 GB file successfully written to disk
  </code></pre>
  
  <p>now here's some <code>RCurl</code> code that does not work.  as stated in <a href="https://stackoverflow.com/questions/17306695/httr-get-function-running-out-of-space-when-downloading-a-large-file">the previous question</a>, reproducing this exactly will require creating an extract on <a href="http://www.ipums.org" rel="nofollow noreferrer">ipums</a>.</p>
  
  <pre><code>your.email &lt;- "email@address.com"
  your.password &lt;- "password"
  extract.path &lt;- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz"
  
  library(RCurl)
  
  values &lt;- 
      list(
          "login[email]" = your.email , 
          "login[password]" = your.password , 
          "login[is_for_login]" = 1
      )
  
  curl = getCurlHandle()
  
  curlSetOpt(
      cookiejar = 'cookies.txt', 
      followlocation = TRUE, 
      autoreferer = TRUE, 
      ssl.verifypeer = FALSE,
      curl = curl
  )
  
  params &lt;- 
      list(
          "login[email]" = your.email , 
          "login[password]" = your.password , 
          "login[is_for_login]" = 1
      )
  
  html &lt;- postForm("https://usa.ipums.org/usa-action/users/validate_login", .params = params, curl = curl)
  dl &lt;- getURL( "https://usa.ipums.org/usa-action/extract_requests/download" , curl = curl)
  </code></pre>
  
  <p>and now that i'm logged in, try the same commands as above, but with the <code>curl</code> object to keep the cookies.</p>
  
  <pre><code>filename &lt;- tempfile()
  f &lt;- CFILE(filename, mode = "wb")
  </code></pre>
  
  <p>this line breaks--</p>
  
  <pre><code>curlPerform(url = extract.path, writedata = f@ref, curl = curl)
  close(f)
  
  # the error is:
  Error in curlPerform(url = extract.path, writedata = f@ref, curl = curl) : 
    embedded nul in string: [[binary jibberish here]]
  </code></pre>
  
  <p>the answer to my previous post referred me to <a href="https://stackoverflow.com/questions/15456317/create-a-c-level-file-handle-in-rcurl-for-writing-downloaded-files/15463343#15463343">this c-level writefunction</a> answer, but i'm clueless about how to re-create that curl_writer C program (on windows?)..</p>
  
  <pre><code>dyn.load("curl_writer.so")
  writer &lt;- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
  curlPerform(URL=url, writefunction=writer)
  </code></pre>
  
  <p>..or why it's even necessary, given that the five lines of code at the top of this question work without anything crazy like <code>getNativeSymbolInfo</code>.  i just don't understand why passing in that extra <code>curl</code> object that stores the authentication/cookies and tells it not to verify SSL would cause code that otherwise works.. to break?</p>