title: spark RDD union action very slow
body: <p>I have 2 spark RDDs, dataRDD and newPairDataRDD which are used for spark SQL query.
  when my application init, the dataRDD will be initialized. All data in one specified hbase entity will be stored to dataRDD.</p>
  
  <p>When client's sql query comes, my APP will get all the new updates and inserts to newPairDataRDD.
  the dataRDD union the newPairDataRDD and register as table in spark SQL context.</p>
  
  <p>I found even 0 record in dataRDD and 1 new inserted record in newPairDataRDD. It will takes 4 seconds for union. That's too slow</p>
  
  <p>I think it is not reasonable. Anyone knows how to make it quicker? Thanks
  simple code as below</p>
  
  <pre><code>    // Step1: load all data from hbase to dataRDD when initial, this only run once. 
      JavaPairRDD&lt;String, Row&gt;  dataRDD= getAllBaseDataToJavaRDD();
      dataRDD.cache();
      dataRDD.persist(StorageLevel.MEMORY_ONLY());
      logger.info(dataRDD.count());
  
      // Step2: when spark sql query coming, load latest updated and inserted data from db to newPairDataRDD
  
      JavaPairRDD&lt;String, Row&gt; newPairDataRDD = getUpdateOrInstertBaseDataToJavaRDD();
      // Step3: if count&gt;0 do union and reduce
  
         if(newPairDataRDD.count() &gt; 0) {
  
          JavaPairRDD&lt;String, Row&gt; unionedRDD =dataRDD.union(newPairDataRDD);
  
      // if data was updated in DB, need to delete the old version from the dataRDD.
  
          dataRDD = unionedRDD.reduceByKey(
              new Function2&lt;Row, Row, Row&gt;() {
              // @Override
              public Row call(Row r1, Row r2) {
               return r2;
               }
              });
      }
  //step4: register the dataRDD
  JavaSchemaRDD schemaRDD = sqlContext.applySchema(dataRDD..values(), schema);
  
  //step5: execute sql query
  retRDD = sqlContext.sql(sql);
  List&lt;org.apache.spark.sql.api.java.Row&gt; rows = retRDD.collect();
  </code></pre>
  
  <p>From the spark web ui, I can see below. Apparently it needs 4s for union</p>
  
  <p>Completed Stages (8)</p>
  
  <p>StageId Description Submitted   Duration Tasks: Succeeded/Total Input Shuffle Read Shuffle Write</p>
  
  <p>6 collect at SparkPlan.scala:85+details 1/4/2015 8:17   2 s 8-Aug       156.0 B </p>
  
  <p>7 union at SparkSqlQueryForMarsNew.java:389+details 1/4/2015 8:17   4 s 8-Aug   64.0 B      156.0 B</p>