title: Memory problems trying to convert a large csv file to a raster layer using R
body: <p>I have a large CSV file representation of a geographic raster map - 42011 columns by 18510 rows, around 10GB on disk. The file was created as the output of a convolution using the FFT function, which involved breaking the original raster into six strips, processing each and appending the resulting matrix to the csv file using the overlap-add method. </p>
  
  <p>I am now trying to write the results back to a raster file type (e.g. ESRI bil, GeoTIFF etc) using the R package <a href="http://cran.r-project.org/web/packages/raster/index.html" rel="nofollow"> raster</a>. Each number in the CSV file maps to the equivalent cell of the raster.</p>
  
  <p>The obvious way to do this in R is:</p>
  
  <pre><code>library(raster)
  library(data.table)
  data &lt;- fread("data.csv")                                  ## Read in data (as data.table)
  data_mat &lt;- data.matrix(data)                              ## Convert to matrix 
  data_raster &lt;- raster(data_mat, template=existing_raster)  ## Write to raster using existing layer as template
  </code></pre>
  
  <p>However, my PC runs out of memory at the stage of trying to create the matrix:</p>
  
  <pre><code>Error: cannot allocate vector of size 5.8 Gb
  In addition: Warning messages:
  1: In matrix(if (intOK) NA_integer_ else NA_real_, nrow = d[1L], ncol = d[2L],  :
     Reached total allocation of 16108Mb: see help(memory.size)
  etc
  </code></pre>
  
  <p>The computer is running 64-bit R with 16GB of RAM.</p>
  
  <p>Things I've tried so far include:</p>
  
  <ul>
  <li><p><strong>bigmemory</strong> - this looked promising, as the raster function can take a big.matrix as it's input, but the package is not currently available for windows.</p></li>
  <li><p><strong>ff</strong> - I haven't fully grasped the capabilities of this package so far, but at present I can't see a way to get from the data.table to an ff object, and I'm not sure there would then be a way to get from that to a raster layer.</p></li>
  <li><p><strong>Sparse matrices</strong> (using package "Matrix") - the CSV is probably more than 50% 0s. However, I can't currently work out a way to read it from disk as a sparse matrix, or to get it from a data.table into a sparse matrix. Whether I would then be able to get to a raster, I have no idea.</p></li>
  </ul>
  
  <p>I am most familiar with R, but also have a working knowledge of Python and access to ArcGIS. I'm not sure if the question is better here or on gis.stackexchange, but it seems like primarily a programming problem. Any suggestions gratefully received!</p>