title: How can I process chunks of data in Spark dataframe
body: <p>Since my data is really huge in TB's I am trying to process and write in smaller chunks in the below fashion but I don't see any result in s3</p>
  <pre><code>val df1 = Seq((1,&quot;Jill&quot;),(2, &quot;John&quot;)).toDF(&quot;id&quot;,&quot;name&quot;)
  val df2 = Seq((1,&quot;accounts&quot;),(2, &quot;finance&quot;)).toDF(&quot;id&quot;,&quot;dept&quot;)
  
  def joinDataSets(row: Row, df2: DataFrame): Unit = {
          val df1 = Seq((row.getAs[java.lang.String](&quot;id&quot;), row.getAs[java.lang.String](&quot;name&quot;))).toDF(&quot;id&quot;, &quot;name&quot;)
          df1.join(df2, df1(&quot;id&quot;) === df2(&quot;id&quot;), &quot;left_outer&quot;)
            .select(&quot;*&quot;)
            .write
            .mode(SaveMode.Append)
            .option(&quot;compression&quot;, &quot;snappy&quot;)
            .parquet(f&quot;s3://{bucket}/test/data&quot;)
        }
  
  df1.rdd.mapPartitions{
        partition =&gt; {
          partition.map(row=&gt;{
            joinDataSets(row, df2)
          })
        }
      }
  </code></pre>
  <p>How can I make this work?
  Note: both the data sets are huge.</p>