title: How to connect HBase and Spark using Python?
body: <p>I have an embarrassingly parallel task for which I use Spark to distribute the computations. These computations are in Python, and I use PySpark to read and preprocess the data. The input data to my task is stored in HBase. Unfortunately, I've yet to find a satisfactory (i.e., easy to use and scalable) way to read/write HBase data from/to Spark using Python.</p>
  
  <p>What I've explored previously:</p>
  
  <ul>
  <li><p>Connecting from within my Python processes using <code>happybase</code>. This package allows connecting to HBase from Python by using HBase's Thrift API. This way, I basically skip Spark for data reading/writing and am missing out on potential HBase-Spark optimizations. Read speeds seem reasonably fast, but write speeds are slow. This is currently my best solution.</p></li>
  <li><p>Using SparkContext's <code>newAPIHadoopRDD</code> and <code>saveAsNewAPIHadoopDataset</code> that make use of HBase's MapReduce interface. Examples for this were once included in the Spark code base (<a href="https://github.com/apache/spark/blob/ed9d80385486cd39a84a689ef467795262af919a/examples/src/main/python/hbase_inputformat.py" rel="noreferrer">see here</a>). However, these are now considered outdated in favor of HBase's Spark bindings (<a href="https://github.com/apache/spark/commit/a680562a6f87a03a00f71bad1c424267ae75c641" rel="noreferrer">see here</a>). I've also found this method to be slow and cumbersome (for reading, writing worked well), for example as the strings returned from <code>newAPIHadoopRDD</code> had to be parsed and transformed in various ways to end up with the Python objects I wanted. It also only supported a single column at a time.</p></li>
  </ul>
  
  <p>Alternatives that I'm aware of:</p>
  
  <ul>
  <li><p>I'm currently using Cloudera's CDH and version 5.7.0 offers <code>hbase-spark</code> (<a href="http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_57.html#concept_hzs_qzh_cv" rel="noreferrer">CDH release notes</a>, and <a href="http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/" rel="noreferrer">a detailed blog post</a>). This module (formerly known as <code>SparkOnHBase</code>) will officially be a part of HBase 2.0. Unfortunately, this wonderful solution seems to work only with Scala/Java.</p></li>
  <li><p>Huawei's <a href="https://github.com/Huawei-Spark/Spark-SQL-on-HBase" rel="noreferrer">Spark-SQL-on-HBase</a> / <a href="https://github.com/HuaweiBigData/astro" rel="noreferrer">Astro</a> (I don't see a difference between the two...). It does not look as robust and well-supported as I'd like my solution to be.</p></li>
  </ul>