title: Spark: Transforming multiple dataframes in parallel
body: <p>Understanding how to achieve best parallelism while transforming multiple dataframes in parallel</p>
  
  <p>I have an array of paths</p>
  
  <pre class="lang-scala prettyprint-override"><code>val paths = Array("path1", "path2", .....
  </code></pre>
  
  <p>I am loading dataframe from each path then transforming and writing to destination path</p>
  
  <pre class="lang-scala prettyprint-override"><code>paths.foreach(path =&gt; {
    val df = spark.read.parquet(path)
    df.transform(processData).write.parquet(path+"_processed")
  })
  </code></pre>
  
  <p>The transformation <code>processData</code> is independent of dataframe I am loading.</p>
  
  <p>This limits to processing one dataframe at a time and most of my cluster resources are idle. As processing each dataframe is independent, I converted <code>Array</code> to <code>ParArray</code> of scala.</p>
  
  <pre class="lang-scala prettyprint-override"><code>paths.par.foreach(path =&gt; {
    val df = spark.read.parquet(path)
    df.transform(processData).write.parquet(path+"_processed")
  })
  </code></pre>
  
  <p>Now it is using more resources in cluster. I am still trying to understand how it works and how to fine tune the parallel processing here</p>
  
  <ol>
  <li><p>If I increase the default scala parallelism using <code>ForkJoinPool</code> to higher number, can it lead to more threads spawning at driver side and will be in lock state waiting for <code>foreach</code> function to finish and eventually kill the driver?</p></li>
  <li><p>How does it effect the centralized spark things like <code>EventLoggingListnener</code> which needs to handle more inflow of events as multiple dataframes are processed in parallel.</p></li>
  <li><p>What parameters do I consider for optimal resource utilization.</p></li>
  <li><p>Any other approach</p></li>
  </ol>
  
  <p>Any resources I can go through to understand this scaling would be very helpful</p>