title: NotServingRegionException: Region hbase:meta
body: <p>We have HBase service installed through Cloudera Manager (5.4), but for some reason we decided to delete the service &amp; redeploy HBase service again. But after redeployment HBase master fails to start with following  </p>
  
  <pre><code>2015-05-13 14:17:56,242 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 73755 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
  2015-05-13 14:17:56,809 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=hdp015.abc.com,60020,1431541075296
  2015-05-13 14:17:56,815 INFO org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
  2015-05-13 14:17:56,844 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 74358 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
  2015-05-13 14:17:58,349 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for region servers count to settle; checked in 1, slept for 75863 ms, expecting minimum of 1, maximum of 2147483647, master is running.
  2015-05-13 14:17:58,355 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://magnetic-hadoop/hbase/WALs/hdp015.abc.com,60020,1431541075296 belongs to an existing region server
  2015-05-13 14:17:58,462 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=hdp015.abc.com,60020,1431540681831, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hdp015.abc.com,60020,1431541075296
  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2761)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4256)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3623)
  at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96)
  at java.lang.Thread.run(Thread.java:745)
  
  2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log dir for server hdp015.abc.com,60020,1431540681831 does not exist
  2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [hdp015.abc.com,60020,1431540681831]
  2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 0 logs in []
  2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [] in 0ms
  2015-05-13 14:17:58,469 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in ZooKeeper
  2015-05-13 14:17:58,531 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to hdp015.abc.com,60020,1431541075296
  2015-05-13 14:17:58,532 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OFFLINE, ts=1431541078483, server=null} to {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296}
  2015-05-13 14:17:58,571 INFO org.apache.hadoop.hbase.master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
  2015-05-13 14:17:58,588 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296}
  2015-05-13 14:17:58,865 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPEN, ts=1431541078865, server=hdp015.abc.com,60020,1431541075296}
  2015-05-13 14:17:58,868 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED of 1588230740 from hdp015.abc.com,60020,1431541075296; deleting unassigned node
  2015-05-13 14:17:58,882 INFO org.apache.hadoop.hbase.master.RegionStates: Onlined 1588230740 on hdp015.abc.com,60020,1431541075296
  2015-05-13 14:17:58,884 INFO org.apache.hadoop.hbase.master.HMaster: hbase:meta assigned=1, rit=false, location=hdp015.abc.com,60020,1431541075296
  2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
  2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: META already up-to date with PB serialization
  2015-05-13 14:17:58,987 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
  2015-05-13 14:17:58,996 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Start to scan the hbase:meta for the current region assignment snappshot
  2015-05-13 14:17:59,004 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Finished to scan the hbase:meta for the current region assignmentsnapshot
  2015-05-13 14:17:59,023 INFO org.apache.hadoop.hbase.master.TableNamespaceManager: Namespace table not found. Creating...
  2015-05-13 14:17:59,063 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
  2015-05-13 14:17:59,064 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.org.apache.hadoop.hbase.TableExistsException: hbase:namespace
  at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
  at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
  at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
  at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1069)
  at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:942)
  at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:613)
  at java.lang.Thread.run(Thread.java:745)
  2015-05-13 14:17:59,076 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
  2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: Stopping server on 60000
  2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.CatalogJanitor: CatalogJanitor-hnn002:60000 exiting
  2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=60000: stopping
  2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopped
  2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopping
  2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.ClusterStatusChore: hnn002.abc.com,60000,1431540999332-ClusterStatusChore exiting
  2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.BalancerChore: hnn002.abc.com,60000,1431540999332-BalancerChore exiting
  2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer
  2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.HFileCleaner: master:hnn002:60000.archivedHFileCleaner exiting
  2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.LogCleaner: master:hnn002:60000.oldLogCleaner exiting
  2015-05-13 14:17:59,081 INFO org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x44d20db9b2a4642, quorum=hdp015.abc.com:2181,hdp005.abc.com:2181,hdp010.abc.com:2181, baseZNode=/hbase
  2015-05-13 14:17:59,083 INFO org.mortbay.log: Stopped HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
  2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ZooKeeper: Session: 0x44d20db9b2a4642 closed
  2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  2015-05-13 14:17:59,189 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14d20db9a0f470c
  2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470c closed
  2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  2015-05-13 14:17:59,248 INFO org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor: hnn002.abc.com,60000,1431540999332.splitLogManagerTimeoutMonitor exiting
  2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470b closed
  2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
  2015-05-13 14:17:59,305 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
  2015-05-13 14:17:59,306 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
  at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
  at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2822)
  </code></pre>