title: How to run an lapply in parallel in R?
body: <p>So I have tried a few different ways of doing this but each returns an a different error which is making me question if i'm even doing it correctly.</p>
  <p>So without any parallel components, we have the following:</p>
  <pre><code>all_necks &lt;- lapply(b_list, b_fun)
  </code></pre>
  <p>This works perfectly; <code>b_list</code> is a dataframe and <code>b_fun</code> is a ton of joins and functions which are to be done on the list.</p>
  <p>Because each run takes about 5 minutes and there are 550 elements in <code>b_list</code>, I need this to be faster to be practical.</p>
  <p>I try <code>future.lapply</code> but get the following error:</p>
  <pre><code>library(future.apply)
  options(future.globals.maxSize= 178258920000)
  plan(multiprocess, workers = 5) ## Parallelize using five cores
  all_necks &lt;- future_lapply(b_list, b_fun)
  
  ERROR:
  Error in serialize(data, node$con) : error writing to connection
  </code></pre>
  <p>Then I tried <code>foreach</code> and got the following:</p>
  <pre><code>library(doParallel) 
  cl &lt;- makeCluster(detectCores())
  registerDoParallel(cl)
  all_necks &lt;- foreach(i = 1:b_list %dopar% {b_fun})
  
  
  ERROR:
  There were 16 warnings (use warnings() to see them)
  1: In environment() : closing unused connection 19 (&lt;-DESKTOP-XXX)
  2: In environment() : closing unused connection 18 (&lt;-DESKTOP-XXX)
  ...
  </code></pre>
  <p>I must be doing this incorrectly but I really just want this long lapply to run faster via parallel processing.</p>
  <p>I would prefer to do this on 5 cores.</p>
  <p>EDIT: Session Info Added</p>
  <pre><code>R version 4.0.1 (2020-06-06)
  Platform: x86_64-w64-mingw32/x64 (64-bit)
  Running under: Windows 10 x64 (build 18363)
  
  Matrix products: default
  
  locale:
  [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
  [4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
  
  attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     
  
  other attached packages:
   [1] future.apply_1.5.0  future_1.17.0       formattable_0.2.0.1 lubridate_1.7.9     data.table_1.12.8   chron_2.3-55       
   [7] Nmisc_0.3.5         anytime_0.3.7       forcats_0.5.0       stringr_1.4.0       dplyr_1.0.0         purrr_0.3.4        
  [13] readr_1.3.1         tidyr_1.1.0         tibble_3.0.1        ggplot2_3.3.2       tidyverse_1.3.0     jsonlite_1.6.1     
  
  loaded via a namespace (and not attached):
   [1] Rcpp_1.0.4.6      lattice_0.20-41   listenv_0.8.0     assertthat_0.2.1  digest_0.6.25     R6_2.4.1         
   [7] cellranger_1.1.0  backports_1.1.7   reprex_0.3.0      evaluate_0.14     httr_1.4.1        pillar_1.4.4     
  [13] rlang_0.4.6       readxl_1.3.1      rstudioapi_0.11   furrr_0.1.0       blob_1.2.1        rmarkdown_2.3    
  [19] htmlwidgets_1.5.1 munsell_0.5.0     tinytex_0.24      broom_0.5.6       compiler_4.0.1    modelr_0.1.8     
  [25] xfun_0.14         pkgconfig_2.0.3   globals_0.12.5    htmltools_0.5.0   tidyselect_1.1.0  codetools_0.2-16 
  [31] fansi_0.4.1       crayon_1.3.4      dbplyr_1.4.4      withr_2.2.0       rappdirs_0.3.1    grid_4.0.1       
  [37] nlme_3.1-148      gtable_0.3.0      lifecycle_0.2.0   DBI_1.1.0         magrittr_1.5      scales_1.1.1     
  [43] cli_2.0.2         stringi_1.4.6     fs_1.4.1          xml2_1.3.2        ellipsis_0.3.1    generics_0.0.2   
  [49] vctrs_0.3.1       tools_4.0.1       glue_1.4.1        hms_0.5.3         parallel_4.0.1    colorspace_1.4-1 
  [55] rvest_0.3.5       knitr_1.28        haven_2.3.1     
  
  
  </code></pre>