title: unable to communicate with hbase through spark
body: <p>I have a project where I need to configure spark and hbase in a local environment. I downloaded spark-2.2.1, hadoop 2.7 and hbase 1.1.8 and configured accordingly on standalone single node Ubuntu 14.04 OS.
  I am able to pull and push data from spark to HDFS but not with hbase. </p>
  
  <p>core-site.xml:  </p>
  
  <pre><code>   &lt;?xml version="1.0"?&gt;
       &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
       &lt;configuration&gt;
           &lt;property&gt;
               &lt;name&gt;fs.defaultFS&lt;/name&gt;
               &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
           &lt;/property&gt;
       &lt;/configuration&gt;
  </code></pre>
  
  <p>hdfs-site.xml:</p>
  
  <pre><code> [root@localhost conf]# cat hdfs-site.xml &lt;?xml version="1.0"?&gt;
   &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
   &lt;configuration&gt;
       &lt;property&gt;
           &lt;name&gt;dfs.replication&lt;/name&gt;
           &lt;value&gt;1&lt;/value&gt;
       &lt;/property&gt;
       &lt;property&gt;
           &lt;name&gt;dfs.namenode.rpc-bind-host&lt;/name&gt;
           &lt;value&gt;0.0.0.0&lt;/value&gt;
       &lt;/property&gt;
      &lt;property&gt;
           &lt;name&gt;dfs.namenode.servicerpc-bind-host&lt;/name&gt;
           &lt;value&gt;0.0.0.0&lt;/value&gt;
       &lt;/property&gt; &lt;/configuration&gt;
  </code></pre>
  
  <p>spark-env.sh</p>
  
  <pre><code>     [root@localhost conf]# cat spark-env.sh
       export JAVA_HOME=/usr/lib/jvm/java-8-oracle
       export SPARK_WORKER_MEMORY=1g
       export SPARK_WORKER_INSTANCES=1
       export SPARK_MASTER_IP=127.0.0.1
       export SPARK_MASTER_PORT=7077
       export SPARK_WORKER_DIR=/app/spark/tmp
  
       # Options read in YARN client mode
  
       export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
       export SPARK_EXECUTOR_INSTANCES=1
       export SPARK_EXECUTOR_CORES=1
       export SPARK_EXECUTOR_MEMORY=1G
       export SPARK_DRIVER_MEMORY=1G
       export SPARK_YARN_APP_NAME=Spark
       export SPARK_CLASSPATH=/opt/hbase/lib/*
  </code></pre>
  
  <p>hbase-site.xml:</p>
  
  <pre><code> [root@localhost conf]# cat hbase-site.xml &lt;?xml version="1.0"?&gt;
   &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
   &lt;configuration&gt;   &lt;property&gt;
       &lt;name&gt;hbase.rootdir&lt;/name&gt;
       &lt;value&gt;hdfs://localhost:9000/hbase&lt;/value&gt;   &lt;/property&gt;   &lt;property&gt;
       &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
       &lt;value&gt;true&lt;/value&gt;   &lt;/property&gt;   &lt;property&gt;
       &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
       &lt;value&gt;localhost&lt;/value&gt;   &lt;/property&gt;
  
      &lt;property&gt;
         &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
         &lt;value&gt;hdfs://localhost:9000/zookeeper&lt;/value&gt;    &lt;/property&gt;    &lt;property&gt;
         &lt;name&gt;hbase.master.dns.interface&lt;/name&gt;
         &lt;value&gt;default&lt;/value&gt;    &lt;/property&gt;   &lt;property&gt;
       &lt;name&gt;hbase.master.ipc.address&lt;/name&gt;
       &lt;value&gt;localhost&lt;/value&gt;   &lt;/property&gt;   &lt;property&gt;
       &lt;name&gt;hbase.regionserver.dns.interface&lt;/name&gt;
       &lt;value&gt;default&lt;/value&gt;   &lt;/property&gt;   &lt;property&gt;
       &lt;name&gt;hbase.regionserver.ipc.address&lt;/name&gt;
       &lt;value&gt;HOSTNAME&lt;/value&gt;   &lt;/property&gt;
  
     &lt;property&gt;
       &lt;name&gt;hbase.zookeeper.dns.interface&lt;/name&gt;
       &lt;value&gt;default&lt;/value&gt;   &lt;/property&gt;
  
   &lt;/configuration&gt;
  </code></pre>
  
  <p>spark-defaults.conf:</p>
  
  <pre><code>  [root@localhost conf]# cat spark-defaults.conf 
       spark.master           
       spark://127.0.0.1:7077 spark.yarn.dist.files
       /opt/spark/conf/hbase-site.xml
  </code></pre>
  
  <p>Errors:
  Even hbase lib(jars) are exported in spark-env.sh it is unable to import hbase libraries (Ex: HBaseConfiguration).</p>
  
  <pre><code>scala&gt; import org.apache.hadoop.hbase.HBaseConfiguration
  &lt;console&gt;:23: error: object hbase is not a member of package org.apache.hadoop
         import org.apache.hadoop.hbase.HBaseConfiguration
                                  ^
  </code></pre>
  
  <p>If I load these jar through --drive-class-path</p>
  
  <pre><code> spark-shell --master local --driver-class-path=/opt/hbase/lib/*
  </code></pre>
  
  <hr>
  
  <pre><code>scala&gt;     conf.set("hbase.zookeeper.quorum","localhost")
  
  scala&gt;     conf.set("hbase.zookeeper.property.clientPort", "2181")
  
  scala&gt;     val connection: Connection = ConnectionFactory.createConnection(conf)
  connection: org.apache.hadoop.hbase.client.Connection = hconnection-0x2a4cb8ae
  
  scala&gt;     val tableName = connection.getTable(TableName.valueOf("employee"))
  tableName: org.apache.hadoop.hbase.client.Table = employee;hconnection-0x2a4cb8ae
  
  scala&gt;     val insertData = new Put(Bytes.toBytes("1"))
  insertData: org.apache.hadoop.hbase.client.Put = {"totalColumns":0,"row":"1","families":{}}
  
  scala&gt;
       |     insertData.addColumn(Bytes.toBytes("emp personal data "), Bytes.toBytes("Name"), Bytes.toBytes("Jeevan"))
  res3: org.apache.hadoop.hbase.client.Put = {"totalColumns":1,"row":"1","families":{"emp personal data ":[{"qualifier":"Name","v
  n":6,"tag":[],"timestamp":9223372036854775807}]}}
  
  scala&gt;     insertData.addColumn(Bytes.toBytes("emp personal data "), Bytes.toBytes("City"), Bytes.toBytes("San Jose"))
  res4: org.apache.hadoop.hbase.client.Put = {"totalColumns":2,"row":"1","families":{"emp personal data ":[{"qualifier":"Name","v
  n":6,"tag":[],"timestamp":9223372036854775807},{"qualifier":"City","vlen":8,"tag":[],"timestamp":9223372036854775807}]}}
  
  scala&gt;     insertData.addColumn(Bytes.toBytes("emp personal data "), Bytes.toBytes("Company"), Bytes.toBytes("Cisco"))
  res5: org.apache.hadoop.hbase.client.Put = {"totalColumns":3,"row":"1","families":{"emp personal data ":[{"qualifier":"Name","v
  n":6,"tag":[],"timestamp":9223372036854775807},{"qualifier":"City","vlen":8,"tag":[],"timestamp":9223372036854775807},{"qualifi
  ":"Company","vlen":5,"tag":[],"timestamp":9223372036854775807}]}}
  
  scala&gt;     insertData.addColumn(Bytes.toBytes("emp personal data "), Bytes.toBytes("location"), Bytes.toBytes("San Jose"))
  res6: org.apache.hadoop.hbase.client.Put = {"totalColumns":4,"row":"1","families":{"emp personal data ":[{"qualifier":"Name","v
  n":6,"tag":[],"timestamp":9223372036854775807},{"qualifier":"City","vlen":8,"tag":[],"timestamp":9223372036854775807},{"qualifi
  ":"Company","vlen":5,"tag":[],"timestamp":9223372036854775807},{"qualifier":"location","vlen":8,"tag":[],"timestamp":9223372036
  4775807}]}}
  </code></pre>
  
  <p>but I dont see any new column in Hbase.</p>
  
  <p>Can any one help please. any reference to configuration would be great. do i need to configure any zookeeper? appreciate your help.  </p>