title: How to convert DataFrame to javaRdd with distibuted copy?
body: <p>I'm new to spark optimizaiton.
  I'm trying to read hive data into a dataFrame. Then I'm converting the dataFrame to javaRdd and running a map function on top of it.
  The problem I'm facing is, the transformation running on top of javaRdd is running with single task. Also the transformations running on top of this javaRdd is running with single task. To parallelize it, I've repartitioned the javaRdd. Is there any better way to do it, since repartition takes more time to shuffle data.</p>
  
  <pre><code>        DataFrame tempDf = df.sqlContext().sql("SELECT * FROM my_table");
  
  // without repartition, the next transformation will run with 1 task only.
          JavaRDD&lt;IMSSummaryPOJO&gt; inputData = tempDf.toJavaRDD().flatMap(new FlatMapFunction&lt;Row, IMSSummaryPOJO&gt;() {
          //map operation
          }).repartition(repartition);
  
  // Even though i've extra executors, if the previous transformation(inputData) is not repartitioned, then this transformation runs with single task.
      JavaPairRDD&lt;Text,IMSMetric&gt; inputRecordRdd = inputData.flatMapToPair(new IMSInputRecordFormat(dimensionName,hllCounterPValue,hllCounterKValue,dimensionConfigMapBroadCast));
  </code></pre>