title: Loading a large Hbase table into SPARK RDD takes long time
body: <p>I am trying to load a large Hbase table into SPARK RDD to run a SparkSQL query on the entity. For an entity with about 6 million rows, it will take about 35 seconds to load it to RDD. Is it expected? Is there any way to shorten the loading process? I have been getting some tips from <a href="http://hbase.apache.org/book/perf.reading.html" rel="nofollow">http://hbase.apache.org/book/perf.reading.html</a> to speed up the process, e.g.,
  scan.setCaching(cacheSize) and only add the necessary attributes/column to scan. 
  I am just wondering if there are other ways to improve the speed?</p>
  
  <p>Here is the code snippet:</p>
  
  <pre><code>SparkConf sparkConf = new SparkConf().setMaster("spark://url").setAppName("SparkSQLTest");
  JavaSparkContext jsc = new JavaSparkContext(sparkConf);
  Configuration hbase_conf = HBaseConfiguration.create();
  hbase_conf.set("hbase.zookeeper.quorum","url");
  hbase_conf.set("hbase.regionserver.port", "60020");
  hbase_conf.set("hbase.master", "url");
  hbase_conf.set(TableInputFormat.INPUT_TABLE, entityName);
  Scan scan = new Scan();
  scan.addColumn(Bytes.toBytes("MetaInfo"), Bytes.toBytes("col1"));
  scan.addColumn(Bytes.toBytes("MetaInfo"), Bytes.toBytes("col2"));
  scan.addColumn(Bytes.toBytes("MetaInfo"), Bytes.toBytes("col3"));
  scan.setCaching(this.cacheSize);
  hbase_conf.set(TableInputFormat.SCAN, convertScanToString(scan));
  JavaPairRDD&lt;ImmutableBytesWritable, Result&gt; hBaseRDD 
  = jsc.newAPIHadoopRDD(hbase_conf,
              TableInputFormat.class, ImmutableBytesWritable.class,
              Result.class);
  logger.info("count is " + hBaseRDD.cache().count());    
  </code></pre>