title: ROracle Connection on Worker Nodes // Automated reporting with R Markdown
body: <p>I am running into several distinct problems while trying to speed up some automated report generation for a large dataset. I'm utilizing R + markdown -> HTML to generate a report, and loop over ~10K distinct groupings for the report accessing the data from Oracle. </p>
  
  <p>The system is comprised mainly of two parts</p>
  
  <ol>
  <li>a main script</li>
  <li>a markdown template file</li>
  </ol>
  
  <p>The main script sets up the computing environment and parallel processing backends:</p>
  
  <pre><code>library(ROracle)
  library(doParallel) ..etc
  ....
  cl &lt;- makeCluster(4)
  clusterEvalQ(cl, con&lt;-dbConnect(db,un,pw)) ##pseudocode...
  </code></pre>
  
  <p>Here the first issue appears to arise. R throws an exception stating the connections on the workers are invalid BUT when I monitor live sessions on Oracle they appear to be fine...</p>
  
  <p>Next, the main calls the loop for report generation. </p>
  
  <pre><code>foreach(i=1:nrow(reportgroups), .packages=c('ROracle', 'ggplot2', 'knitr') %dopar% ##...etc 
  {
      rmarkdown::render(inputfile.Rmd, outputfile.html, params=list(groupParam1[i], groupParam2[i], etc)
  }
  </code></pre>
  
  <p>If I run the foreach loop sequentially i.e., %do% instead of %dopar%, everything seems to work fine. No errors, then entire set runs correctly (I have only tested to ~400 groups, will do a full run of all 10k overnight).</p>
  
  <p>However, if I attempt to run the loop in parallel, invariably 'pandoc' throws an error #1 in converting the file. If I run the broken loop multiple times, the 'task' in the loop (or cluster, not sure which task refers to in this context) which causes the error changes. </p>
  
  <p>The template file is pretty basic, it takes in parameters for groups, runs an SQL query on the connection defined for the cluster worker, and utilizes ggplot2 + dplyr to generate results. Since the template seem to operate when not through a cluster, I believe the problem must be something to do with the connection objects in the cluster nodes from ROracle, although I don't know enough about the subject to really pinpoint the problem. </p>
  
  <p>If anyone has had a similar experience, or has a hunch about what is going on, any advice would be appreciated! </p>
  
  <p>Let me know if I can clarify anything...</p>
  
  <p>Thanks</p>