title: Manual Fix of Hbase table Overlap (Multi region has same start key)
body: <p>I was inserting the data into Hbase through the java client. But, suddenly the Region server crashed at a point. So i restarted the Hbase, which after that the Hmaster was not running. When i run the repair tool - <code>hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair</code>, i got the reason of follows:-</p>
  
  <pre><code>ERROR: (region TABLE_NAME, rowkey_value, region_name_1) Multiple regions have the same start key: rowkey_value 
  ERROR: (region TABLE_NAME, rowkey_value, region_name_2) Multiple regions have the same start key: rowkey_value 
  ERROR: (regions Table_Name, rowkey_value, region_name_1 and Table_Name, rowkey_value, region_name_2) There is an overlap in the region chain. 
  ERROR: Found inconsistency in table TABLE_NAME
  </code></pre>
  
  <p>I cannot run the hbck (online fix/repair) since my Hmaster does not run. I need to fix this manually now. There is huge data now so i do not want to lose them.</p>
  
  <p>EDIT:</p>
  
  <p>I formatted the namenode and added data again and voila, the same issue again!!
  This time, I had the Hmaster running. So i executed the <code>hbck -fixHdfsOverlaps</code> command, but after a long time, i found that now, both Hmaster and Namenode is not running. Now, since NN is also down, i don't have an access to the HDFS.
  I do not want to lose any data. I noticed that one of the region was having lots of data in the ".tmp" directory of its region, wich is yet to be compacted too.
  So, how to resume the HDFS and hbase back again?</p>
  
  <hr>
  
  <p>LOG INFO of NameNode:</p>
  
  <pre><code>2014-07-09 09:19:44,498 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
  org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/hdfstmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
          at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:288)
          at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.&lt;init&gt;(FSNamesystem.java:360)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.java:496)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
  2014-07-09 09:19:44,500 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data/hdfstmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
          at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:288)
          at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.&lt;init&gt;(FSNamesystem.java:360)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.java:496)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
  
  2014-07-09 09:19:44,501 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
  /************************************************************
  SHUTDOWN_MSG: Shutting down NameNode at ip-10-237-184-7.ec2.internal/10.237.184.7
  ************************************************************/
  </code></pre>
  
  <hr>
  
  <p>Then hdfs temp directory contains only the <strong>dfs</strong> and <strong>mapred</strong> directory. the <strong>name</strong> directory is missing. So does the image and previous.checkpoint, edits etc. which are inside the name dir.
  Now, does it mean the data is lost? Or it cannot be recovered?
  Is there a way i can backup the data?</p>
  
  <p>EDIT (2) :
  I see that the data directory is empty after the <code>hbck -fixHdfsOverlaps</code> command is run. Or does the hbck command temporarily moves the data to a different place while running the recovery? If so, where it is and how it can be brought back and resume the HDFS again without any or minimal data loss.</p>