title: R and shared memory for parallel::mclapply
body: <p>I am trying to take advantage of a quad-core machine by parallelizing a costly operation that is performed on a list of about 1000 items.</p>
  
  <p>I am using R's parallel::mclapply function currently:</p>
  
  <pre><code>res = rbind.fill(parallel::mclapply(lst, fun, mc.cores=3, mc.preschedule=T))
  </code></pre>
  
  <p>Which works. Problem is, any additional subprocess that is spawned has to allocate a large chunk of memory:</p>
  
  <p><img src="https://i.stack.imgur.com/SLoEF.png" alt="enter image description here"></p>
  
  <p>Ideally, I would like each core to access shared memory from the parent R process, so that as I increase the number of cores used in mclapply, I don't hit RAM limitations before core limitations.</p>
  
  <p>I'm currently at a loss on how to debug this issue. All of the large data structures that each process accesses are globals (currently). Is that somehow the issue?</p>
  
  <p>I did increase my shared memory max setting for the OS to 20 GB (available RAM):</p>
  
  <pre><code>$ cat /etc/sysctl.conf 
  kern.sysv.shmmax=21474836480
  kern.sysv.shmall=5242880
  kern.sysv.shmmin=1
  kern.sysv.shmmni=32
  kern.sysv.shmseg=8
  kern.maxprocperuid=512
  kern.maxproc=2048
  </code></pre>
  
  <p>I thought that would fix things, but the issue still occurs.</p>
  
  <p>Any other ideas?</p>