title: doParallel (package) foreach does not work for big iterations in R
body: <p>I'm running the following code (extracted from <a href="https://cran.r-project.org/web/packages/doParallel/vignettes/gettingstartedParallel.pdf" rel="nofollow noreferrer">doParallel's Vignettes</a>) on a PC (OS Linux) with 4 and 8 physical and logical cores, respectively. </p>
  
  <p>Running the code with <code>iter=1e+6</code> or less, every thing is fine and I can see from CPU usage that all cores are employed for this computation. However, with larger number of iterations (e.g. <code>iter=4e+6</code>), it seems parallel computing does not work in which case. When I also monitor the CPU usage, just one core is involved in computations (100% usage).</p>
  
  <p><strong>Example1</strong></p>
  
  <pre><code>require("doParallel")
  require("foreach")
  registerDoParallel(cores=8)
  x &lt;- iris[which(iris[,5] != "setosa"), c(1,5)]
  iter=4e+6
  ptime &lt;- system.time({
      r &lt;- foreach(i=1:iter, .combine=rbind) %dopar% {
          ind &lt;- sample(100, 100, replace=TRUE)
          result1 &lt;- glm(x[ind,2]~x[ind,1], family=binomial(logit))
          coefficients(result1)
      }
  })[3]
  </code></pre>
  
  <p>Do you have any idea what could be the reason? Could memory be the cause?</p>
  
  <p>I googled around and I found <a href="https://stackoverflow.com/questions/32067837/foreach-loop-r-doparallel-package-fails-with-big-number-of-iterations">THIS</a> relevant to my question but the point is that I'm not given any kind of error and the OP seemingly has came up with a solution by providing necessary packages inside <code>foreach</code> loop. But no package is used inside my loop, as can be seen.</p>
  
  <p><strong>UPDATE1</strong></p>
  
  <p>My problem still is not solved. As per my experiments, I don't think that memory could be the reason. I have 8GB of memory on the system on which I run the following simple parallel (over all 8 logical cores) iteration:</p>
  
  <p><strong>Example2</strong></p>
  
  <pre><code>require("doParallel")
  require("foreach")
  
  registerDoParallel(cores=8)
  iter=4e+6
  ptime &lt;- system.time({
      r &lt;- foreach(i=1:iter, .combine=rbind) %dopar% {
          i
      }
  })[3]
  </code></pre>
  
  <p>I do not have problem with running of this code but when I monitor the CPU usage, just one core (out of 8) is 100%.</p>
  
  <p><strong>UPDATE2</strong></p>
  
  <p>As for <strong>Example2</strong>, @SteveWeston (thanks for pointing this out) stated that (in comments) : "The example in your update is suffering from having tiny tasks. Only the master has any real work to do, which consists of sending tasks and processing results. That's fundamentally different than the problem with the original example which did use multiple cores on a smaller number of iterations."</p>
  
  <p>However, <strong>Example1</strong> still remains unsolved. When I run it and I monitor the processes with <code>htop</code>, here is what happens in more detail:</p>
  
  <p>Let's name all 8 created processes <code>p1</code> through <code>p8</code>. The status (column <code>S</code> in <code>htop</code>) for <code>p1</code> is <code>R</code> meaning that it's running and remains unchanged. However, for <code>p2</code> up to <code>p8</code>, after some minutes, the status changes to <code>D</code> (i.e. uninterruptible sleep) and, after some minutes, again changes to <code>Z</code> (i.e. terminated but not reaped by its parent). Do you have any idea why this happens?</p>