title: HBase Master cannot be contacted by Region Servers
body: <p>I have a Hadoop cluster made of 3 slaves and 1 master on top of which there is an HBase cluster with 3 RS and 1 master respectively. Additionally there is a Zookeeper ensemble on 3 machines. </p>
  
  <p>The Hadoop cluster is functioning correctly as well as the Zookeeper ensemble. However, the HBase cluster fails to initialize correctly.</p>
  
  <p>I start HBase it by running <code>./bin/start-hbase.sh</code>. This <strong>correctly</strong> starts the HBase Master and the Region Servers. The hbase folder in hdfs is set-up correctly.</p>
  
  <p><strong>jps on master</strong></p>
  
  <pre><code>hduser@master:~/hbase$ jps
  5694 HMaster
  3934 JobHistoryServer
  3786 NameNode
  3873 ResourceManager
  6025 Jps
  </code></pre>
  
  <p><strong>jps on slaves</strong></p>
  
  <pre><code>5737 Jps
  5499 HRegionServer
  3736 DataNode
  3820 NodeManager
  </code></pre>
  
  <p>However, the HBase master does not register the Region Servers as it is also apparent from looking at the logs:</p>
  
  <p><strong>master log</strong></p>
  
  <pre><code>[master:master:60000] master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 1511 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
  </code></pre>
  
  <p><strong>slave log</strong></p>
  
  <pre><code>[regionserver60020] regionserver.HRegionServer: reportForDuty to master=master,60000,1404856451890 with port=60020, startcode=1404856453874
  [regionserver60020] regionserver.HRegionServer: error telling master we are up
  com.google.protobuf.ServiceException: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending local=/10.0.2.15:53939 remote=master/192.168.66.60:60000]
  </code></pre>
  
  <p>Here are the configuration details:</p>
  
  <p><strong>/etc/hosts on master</strong></p>
  
  <pre><code>192.168.66.63   slave-3 # Data Node and Region Server
  192.168.66.60   master # Name Node and HBase Master
  192.168.66.73   zookeeper-3 # Zookeeper node
  192.168.66.71   zookeeper-1 # Zookeeper node
  192.168.66.72   zookeeper-2 # Zookeeper node
  192.168.66.62   slave-2 # Data Node and Region Server
  192.168.66.61   slave-1 # Data Node and Region Server
  </code></pre>
  
  <p><strong>/etc/hosts on slave-1</strong></p>
  
  <pre><code>192.168.66.60   master
  192.168.66.73   zookeeper-3
  192.168.66.71   zookeeper-1
  192.168.66.72   zookeeper-2
  </code></pre>
  
  <p><strong>hbase-site.xml on ALL cluster nodes</strong></p>
  
  <pre><code>&lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  &lt;configuration&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.tmp.dir&lt;/name&gt;
          &lt;value&gt;/home/hduser/hbase/tmp&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.rootdir&lt;/name&gt;
          &lt;value&gt;hdfs://master/hbase&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
          &lt;value&gt;true&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.local.dir&lt;/name&gt;
          &lt;value&gt;/home/hduser/hbase/local&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.master.info.port&lt;/name&gt;
          &lt;value&gt;6010&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
          &lt;value&gt;zookeeper-1,zookeeper-2,zookeeper-3,&lt;/value&gt;
      &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>regionservers file on master and slaves</strong></p>
  
  <pre><code>slave-3
  slave-1
  slave-2
  </code></pre>
  
  <p><strong>hbase-env.sh on master and slaves</strong></p>
  
  <pre><code>export JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:/bin/javac::"
  export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
  export HBASE_MANAGES_ZK=false
  </code></pre>
  
  <p>What am I doing wrong so that the nodes cannot talk to each other? I am using Hadoop 2.4.0 and Hbase 0.98.3 along with Zookeeper 3.4.6 on Ubuntu Trusty Tahr x64.</p>