title: Index large amount of data into elasticsearch
body: <p>I've got more than 6 billion of social media data in HBase (including content/time/author and other possible fields) with 4100 regions in 48 servers , and I need to flush these data into Elasticsearch now.</p>
  
  <p>I'm clear about the bulk API of ES, and using bulk in Java with MapReduce still cost many days (at least a week or so). I can use spark instead but I don't think it will help a lot.</p>
  
  <p>I'm wondering is there any other tricks to write these large data into ElasticSearch ? Like manually write to es index files and using some kinds of recover to load the files in local file system ?</p>
  
  <p>Appreciate any possible advice, thanks.</p>
  
  <p>==============</p>
  
  <p>Some details about my cluster environments:</p>
  
  <p>spark 1.3.1 standalone (I can change it on yarn to use Spark 1.6.2 or 1.6.3)</p>
  
  <p>Hadoop 2.7.1 (HDP 2.4.2.258)</p>
  
  <p>ElasticSearch 2.3.3</p>