title: reading and processing files in parallel in R
body: <p>I am using the <code>parallel</code> library in R to process a large data set on which I am applying complex operations.</p>
  
  <p>For the sake of providing a reproducible code, you can find below a simpler example:</p>
  
  <pre><code>#data generation
  dir &lt;- "C:/Users/things_to_process/"
  
  setwd(dir)
  for(i in 1:800)
  {
      my.matrix &lt;- matrix(runif(100),ncol=10,nrow=10)
  
      saveRDS(my.matrix,file=paste0(dir,"/matrix",i))
  }
  
  #worker function
  worker.function &lt;- function(files)
  {
      files.length &lt;- length(files)
      partial.results &lt;- vector('list',files.length)
  
      for(i in 1:files.length)
      {
          matrix &lt;- readRDS(files[i])
          partial.results[[i]] &lt;- sum(diag(matrix))
      }
  
      Reduce('+',partial.results) 
  }
  
  
  #master part
  cl &lt;- makeCluster(detectCores(), type = "PSOCK")
  
  file_list &lt;- list.files(path=dir,recursive=FALSE,full.names=TRUE)
  
  part &lt;- clusterSplit(cl,seq_along(file_list))
  files.partitioned &lt;- lapply(part,function(p) file_list[p])
  
  results &lt;- clusterApply(cl,files.partitioned,worker.function)
  
  result &lt;- Reduce('+',results)
  </code></pre>
  
  <p>Essentially, I am wondering if trying to read files in parallel would be done in an interleaved fashion instead. And if, as a result, this bottleneck would cut down on the expected performance of running tasks in parallel?</p>
  
  <p>Would it be better if I first read all matrices at once in a list then sent chunks of this list to each core for it to be processed? what if these matrices were much larger, would I be able to load all of them in a list at once ?</p>