title: Fatal error of "failed to become active master" while running hbase in cluster mode
body: <p>I have 4 nodes, one master and 3 slaves.</p>
  
  <p>master: * .* .*.18, slaves: * .*. *.12, 104, 36.</p>
  
  <p>Configurations for Hadoop on Namenode:</p>
  
  <p>core-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
      &lt;name&gt;fs.defaultFS&lt;/name&gt;
      &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>hdfs-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.replication&lt;/name&gt;
      &lt;value&gt;3&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;
      &lt;value&gt;file:///home/hduser/hadoop_store/hdfs/namenode&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;
      &lt;value&gt;file:///home/hduser/hadoop_store/hdfs/datanode&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>hadoop-env.sh:</p>
  
  <pre><code>export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
  export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
  export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
  export HADOOP_PID_DIR=${HADOOP_PID_DIR} // default to /tmp
  export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
  export HADOOP_IDENT_STRING=$USER
  </code></pre>
  
  <p>mapred-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
          &lt;name&gt;mapreduce.framework.name&lt;/name&gt;
          &lt;value&gt;yarn&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
          &lt;name&gt;mapred.job.tracker&lt;/name&gt;
          &lt;value&gt;localhost:54311&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>slaves:</p>
  
  <pre><code>10.0.3.12
  10.0.3.36
  10.0.3.104
  </code></pre>
  
  <p>yarn-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  
  &lt;!-- Site specific YARN configuration properties --&gt;
  
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
      &lt;value&gt;localhost:8050&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
      &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;
      &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p>In the slave nodes the configurations for hadoop are: </p>
  
  <p>yarn-site.xml:</p>
  
  <pre><code>&lt;configuration&gt;
  
  &lt;!-- Site specific YARN configuration properties --&gt;
  
  &lt;property&gt;
      &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
      &lt;value&gt;10.0.3.18:8050&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.address&lt;/name&gt;
      &lt;value&gt;localhost:8035&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
      &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;
      &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p>the rest of the files remain the same in all the slave nodes as in the master node. With respect to the Hbase configuration, </p>
  
  <p>hbase-env.sh(in all):</p>
  
  <pre><code>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"
  export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"
  export HBASE_REGIONSERVERS=${HBASE_HOME}/conf/regionservers
  export HBASE_MANAGES_ZK=true
  </code></pre>
  
  <p>hbase-site.xml(in all):</p>
  
  <pre><code>&lt;configuration&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.rootdir&lt;/name&gt;
              &lt;value&gt;hdfs://localhost:9000/hbase&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.cluster.distributed&lt;/name&gt;
              &lt;value&gt;true&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
              &lt;value&gt;10.0.3.18,10.0.3.12,10.0.3.104,10.0.3.36&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.zookeeper.property.dataDir&lt;/name&gt;
              &lt;value&gt;/home/hduser/Downloads/hbase/zookeeper&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.zookeeper.property.clientPort&lt;/name&gt;
              &lt;value&gt;2181&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;dfs.replication&lt;/name&gt;
              &lt;value&gt;3&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;zookeeper.session.timeout&lt;/name&gt;
              &lt;value&gt;1200000&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
              &lt;name&gt;hbase.zookeeper.property.tickTime&lt;/name&gt;
              &lt;value&gt;6000&lt;/value&gt;
      &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>except that in slaves, localhost is changed to 10.0.3.18(address of namenode)</p>
  
  <p>regionservers:</p>
  
  <pre><code>10.0.3.12
  10.0.3.104
  10.0.3.36
  </code></pre>
  
  <p>I formatted namenode and when I start hdfs with commands: start-dfs.sh and start-yarn.sh, output is as follows: </p>
  
  <pre><code>...succefully formatted namenode...
  localhost: starting namenode, logging to /home/hduser/Downloads/hadoop/logs/hadoop-hduser-namenode-saichanda-OptiPlex-9020.out
  10.0.3.12: starting datanode, logging to /home/hduser/Downloads/hadoop/logs/hadoop-hduser-datanode-aaron.out
  10.0.3.36: starting datanode, logging to /home/hduser/Downloads/hadoop/logs/hadoop-hduser-datanode-dmacs-OptiPlex-9020.out
  10.0.3.104: starting datanode, logging to /home/hduser/Downloads/hadoop/logs/hadoop-hduser-datanode-hadoop-104.out
  Starting secondary namenodes [0.0.0.0]
  0.0.0.0: starting secondarynamenode, logging to /home/hduser/Downloads/hadoop/logs/hadoop-hduser-secondarynamenode-saichanda-OptiPlex-9020.out
  starting yarn daemons
  starting resourcemanager, logging to /home/hduser/Downloads/hadoop/logs/yarn-hduser-resourcemanager-saichanda-OptiPlex-9020.out
  10.0.3.12: starting nodemanager, logging to /home/hduser/Downloads/hadoop/logs/yarn-hduser-nodemanager-aaron.out
  10.0.3.36: starting nodemanager, logging to /home/hduser/Downloads/hadoop/logs/yarn-hduser-nodemanager-dmacs-OptiPlex-9020.out
  10.0.3.104: starting nodemanager, logging to /home/hduser/Downloads/hadoop/logs/yarn-hduser-nodemanager-hadoop-104.out
  </code></pre>
  
  <p>when I run jps command (on master):</p>
  
  <pre><code>28032 SecondaryNameNode
  28481 Jps
  28198 ResourceManager
  27720 NameNode
  </code></pre>
  
  <p>when I run jps command (on slaves):</p>
  
  <pre><code>11303 DataNode
  11595 Jps
  11436 NodeManager
  </code></pre>
  
  <p>Then I started Hbase with the command: ./start-hbase.sh. output is:</p>
  
  <pre><code>10.0.3.12: running zookeeper, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-zookeeper-aaron.out
  10.0.3.36: running zookeeper, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-zookeeper-dmacs-OptiPlex-9020.out
  10.0.3.104: running zookeeper, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-zookeeper-hadoop-104.out
  10.0.3.18: running zookeeper, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-zookeeper-saichanda-OptiPlex-9020.out
  running master, logging to /home/hduser/Downloads/hbase/logs/hbase-hduser-master-saichanda-OptiPlex-9020.out
  OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
  OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  10.0.3.12: running regionserver, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-regionserver-aaron.out
  10.0.3.36: running regionserver, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-regionserver-dmacs-OptiPlex-9020.out
  10.0.3.104: running regionserver, logging to /home/hduser/Downloads/hbase/bin/../logs/hbase-hduser-regionserver-hadoop-104.out
  10.0.3.12: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
  10.0.3.12: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  10.0.3.36: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
  10.0.3.36: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  10.0.3.104: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
  10.0.3.104: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
  </code></pre>
  
  <p>when I run jps on namenode:</p>
  
  <pre><code>28032 SecondaryNameNode
  28821 HQuorumPeer
  29126 Jps
  28198 ResourceManager
  27720 NameNode
  </code></pre>
  
  <p>when I run jps on slaves:</p>
  
  <pre><code>11776 HRegionServer
  11669 HQuorumPeer
  11303 DataNode
  11899 Jps
  11436 NodeManager
  </code></pre>
  
  <p>What I observed was that HMaster is not running on the namenode. Can anyone help understand the problem why HMaster is crashing out. After sometime even NodeManager crashes out in the slaves. Also I observed that When I shutdown hbase, HRegionservers on the slaves donot go down, they continue to be running even after I give stop-hbase.sh command in the master node. Key warnings and errors observed in My logs are as follows.</p>
  
  <p>hadoop-namenode.log: multiple times I get this Exception...</p>
  
  <pre><code>java.io.IOException: File /hbase/.tmp/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
  </code></pre>
  
  <p>hadoop-secondary-namenode.log: multiple times I get this ERROR...</p>
  
  <pre><code>ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
  java.io.IOException: Inconsistent checkpoint fields.
  </code></pre>
  
  <p>No error found in yarn-resourcemanager.log.</p>
  
  <p>For hbase logs: in hbase-master.log:</p>
  
  <pre><code> FATAL [saichanda-OptiPlex-9020:16000.activeMasterManager] master.HMaster: Failed to become active master
  File /hbase/.tmp/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
  </code></pre>
  
  <p>In hbase-zookeeper.log: I see this line, as such no errors were there in the log.</p>
  
  <pre><code>019-01-29 10:09:49,431 INFO  [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
  </code></pre>
  
  <p>on one of the slaves, regionserver.log:</p>
  
  <pre><code> client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
  </code></pre>
  
  <p>on one of the slaves, hadoop-datanode.log gives multiple times the following warning.</p>
  
  <pre><code>WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: localhost/127.0.0.1:9000
  </code></pre>
  
  <p>AMONG ALL THE ABOVE WARNINGS AND ERRORS, I FEEL THE ERROR PERTAINING TO HBASE-MASTER.LOG SEEMS CRITICAL, WHERE IT SAYS, replicated to 0 nodes instead of minReplication (=1). Please help me solve this issue.</p>
  
  <p>Also, when I finally run the hbase shell, I get the error:</p>
  
  <pre><code>ERROR: Can't get master address from ZooKeeper; znode data == null
  </code></pre>
  
  <p>Thank you.</p>