title: Spark DataFrame Parallelism
body: <p>Below is my usecase i am using Apache Spark</p>
  
  <p>1) I have around 2500 Parquet files on HDFS, file size varies from file to file.</p>
  
  <p>2) I need to process each parquet files and build a new DataFrame and write a new DataFrame into orc file format.</p>
  
  <p>3) My Spark driver program is like this.
  I am iterating  each file, processing single parquet file creating a new DataFrame and writing a new DataFrame as ORC, below is the code snippet.</p>
  
  <pre><code>  val fs = FileSystem.get(new Configuration())
    val parquetDFMap = fs.listStatus(new Path(inputFilePath)).map(folder =&gt; {
    (folder.getPath.toString, sqlContext.read.parquet(folder.getPath.toString))})
  
  parquetDFMap.foreach {
    dfMap =&gt;
      val parquetFileName = dfMap._1
      val parqFileDataFrame = dfMap._2
      for (column &lt;- parqFileDataFrame.columns) 
      {
         val rows = parqFileDataFrame.select(column)
              .mapPartitions(lines =&gt; lines.filter(filterRowsWithNullValues(_))
              .map(row =&gt; buildRowRecords(row, masterStructArr.toArray, valuesArr)))
          val newDataFrame: DataFrame = parqFileDataFrame.sqlContext.createDataFrame(rows, StructType(masterStructArr))
         newDataFrame.write.mode(SaveMode.Append).format("orc").save(orcOutPutFilePath+tableName)
      }
  }
  </code></pre>
  
  <p>The problem with this design I am able to process only one parquet file in time, parallelism is applied only when I create a new data frame and when the new DataFrame is written into ORC format. So if any of the tasks like creating a new DataFrame or writing a new DataFrame in to ORC take long time to complete other lined up parquet processing is stuck until the current parquet operation gets completed.</p>
  
  <p>Can you please help me with a better approach or design for this usecase.</p>