title: HBase have no response to client for minutes
body: <p>I have a distributed hbase cluster, with three regionserver and two tables, millions rows</p>
  
  <p>Occasionally the cluster don't response to both read and write requests for about <strong>15 minutes</strong>, this happened several times, and last 15 minutes every time, once in about a week</p>
  
  <p>I have no idea why, and find no error in logs, GC for hbase master and region server is finish in acceptable time, find no useful information after google</p>
  
  <p>Can someone help me, thanks very much</p>
  
  <p>My hbase version is 0.94.18 with hadoop is 1.2.1, and asynchbase 1.4.1 as client</p>
  
  <p>region server log during the 15 minutes with no reponse is list blow:</p>
  
  <pre><code>no response 15 minutes last from 2014-08-31 20:54:45 to 2014-08-31 21:09:16
  ----------------------------------------------------------------------------------------
  
  2014-08-31 20:41:02,488 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc. after a delay of 7351
  2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc., current region memstore size 94.1k
  2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc., syncing WAL and waiting on mvcc, flushsize=96376
  2014-08-31 20:41:09,841 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  2014-08-31 20:41:09,845 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b with permission=rwxrwxrwx
  2014-08-31 20:41:09,846 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 20:41:09,846 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 20:41:09,879 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 20:41:09,880 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b: CompoundBloomFilterWriter
  2014-08-31 20:41:09,909 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b)
  2014-08-31 20:41:09,909 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315537, memsize=93.9k, into tmp file hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b
  2014-08-31 20:41:09,919 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/.tmp/97b6d77913704f56aef1d39127bea09b to hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/a/97b6d77913704f56aef1d39127bea09b
  2014-08-31 20:41:09,930 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/4e8ffbbffd0a125b4001910affec45cc/a/97b6d77913704f56aef1d39127bea09b, entries=8, sequenceid=3315537, filesize=93.7k
  2014-08-31 20:41:09,932 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~94.1k/96376, currentsize=0.0/0 for region bw,,1408098137403.4e8ffbbffd0a125b4001910affec45cc. in 91ms, sequenceid=3315537, compaction requested=false
  2014-08-31 20:44:42,488 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045. after a delay of 6086
  2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045., current region memstore size 37.1k
  2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045., syncing WAL and waiting on mvcc, flushsize=38008
  2014-08-31 20:44:48,575 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  2014-08-31 20:44:48,582 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e with permission=rwxrwxrwx
  2014-08-31 20:44:48,583 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 20:44:48,583 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 20:44:48,601 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 20:44:48,601 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e: CompoundBloomFilterWriter
  2014-08-31 20:44:48,624 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e)
  2014-08-31 20:44:48,624 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315613, memsize=36.9k, into tmp file hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e
  2014-08-31 20:44:48,632 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/.tmp/f45f88b313904de49eec3f702abf811e to hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/a/f45f88b313904de49eec3f702abf811e
  2014-08-31 20:44:48,650 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/24284ecebbdfcd95f153a46beabe0045/a/f45f88b313904de49eec3f702abf811e, entries=8, sequenceid=3315613, filesize=36.6k
  2014-08-31 20:44:48,652 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~37.1k/38008, currentsize=0.0/0 for region bw,01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137403.24284ecebbdfcd95f153a46beabe0045. in 76ms, sequenceid=3315613, compaction requested=false
  2014-08-31 20:45:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=219, accesses=956789177, hits=956375329, hitRatio=99.95%, , cachingAccesses=956377890, cachingHits=956349801, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 20:49:12,491 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a. after a delay of 4475
  2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a., current region memstore size 141.4k
  2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a., syncing WAL and waiting on mvcc, flushsize=144792
  2014-08-31 20:49:16,967 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  2014-08-31 20:49:16,978 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f with permission=rwxrwxrwx
  2014-08-31 20:49:16,979 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 20:49:16,979 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 20:49:17,021 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 20:49:17,021 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f: CompoundBloomFilterWriter
  2014-08-31 20:49:17,051 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f)
  2014-08-31 20:49:17,052 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315721, memsize=141.2k, into tmp file hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f
  2014-08-31 20:49:17,060 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/.tmp/b7b2bdd1f6384466b30a4363c2b7f22f to hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/a/b7b2bdd1f6384466b30a4363c2b7f22f
  2014-08-31 20:49:17,071 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/b97a9d0c413fdddd8a5dc7e82ba59f7a/a/b7b2bdd1f6384466b30a4363c2b7f22f, entries=8, sequenceid=3315721, filesize=141.0k
  2014-08-31 20:49:17,073 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~141.4k/144792, currentsize=0.0/0 for region bw,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.b97a9d0c413fdddd8a5dc7e82ba59f7a. in 106ms, sequenceid=3315721, compaction requested=false
  2014-08-31 20:50:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=220, accesses=956969362, hits=956555513, hitRatio=99.95%, , cachingAccesses=956558075, cachingHits=956529985, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 20:54:32,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 21448
  2014-08-31 20:54:42,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 21023
  2014-08-31 20:54:52,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. after a delay of 17156
  2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f., current region memstore size 140.1k
  2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f., syncing WAL and waiting on mvcc, flushsize=143512
  2014-08-31 20:54:53,943 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  2014-08-31 20:54:53,948 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784 with permission=rwxrwxrwx
  2014-08-31 20:54:53,949 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 20:54:53,949 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 20:54:53,985 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 20:54:53,985 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784: CompoundBloomFilterWriter
  2014-08-31 20:54:54,012 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784)
  2014-08-31 20:54:54,012 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3315851, memsize=140.0k, into tmp file hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784
  2014-08-31 20:54:54,021 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/.tmp/e521dc246e6c4216bbb368f0ce98e784 to hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/a/e521dc246e6c4216bbb368f0ce98e784
  2014-08-31 20:54:54,032 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/bw/13c8d51da8d7ff842bbc8f584843ee9f/a/e521dc246e6c4216bbb368f0ce98e784, entries=8, sequenceid=3315851, filesize=139.7k
  2014-08-31 20:54:54,034 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~140.1k/143512, currentsize=0.0/0 for region bw,04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1408098137404.13c8d51da8d7ff842bbc8f584843ee9f. in 91ms, sequenceid=3315851, compaction requested=false
  2014-08-31 20:55:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=18.95 MB, free=478.97 MB, max=497.92 MB, blocks=221, accesses=957103402, hits=956689552, hitRatio=99.95%, , cachingAccesses=956692115, cachingHits=956664024, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 20:56:27,483 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 2176807346167982283 lease expired on region messages,07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.1a34b2e385e39cf3a82ae55f732c7a4e.
  .......................
  2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -6933263087733055502 lease expired on region messages,11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.198c072806a86f7103a0f3c6de11290a.
  2014-08-31 20:56:38,174 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -2230372302736915424 lease expired on region messages,08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.3ab90beebbc002e19a2f024cb74e5e99.
  2014-08-31 20:59:55,789 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of info because one (major) compacted file only and oldestTime 7395388096ms is &lt; ttl=9223372036854775807
  2014-08-31 21:00:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957113714, hits=956699856, hitRatio=99.95%, , cachingAccesses=956702427, cachingHits=956674328, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 21:09:41,115 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Client tried to access missing scanner -5479515347986367613
  .......................
  2014-08-31 21:10:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957150432, hits=956736574, hitRatio=99.95%, , cachingAccesses=956739145, cachingHits=956711046, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 21:10:41,126 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 2475283676294571093 lease expired on region messages,07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407978.1a34b2e385e39cf3a82ae55f732c7a4e.
  .........................
  2014-08-31 21:10:41,127 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -1230151222882130492 lease expired on region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
  2014-08-31 21:10:41,127 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -3189762669648092281 lease expired on region 
  ...............
  2014-08-31 21:10:41,731 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 4766251358165302168 lease expired on region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
  2014-08-31 21:15:11,191 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=19.41 MB, free=478.52 MB, max=497.92 MB, blocks=229, accesses=957331387, hits=956917529, hitRatio=99.95%, , cachingAccesses=956920100, cachingHits=956892001, cachingHitsRatio=99.99%, , evictions=0, evicted=26865, evictedPerRun=Infinity
  2014-08-31 21:16:32,506 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. after a delay of 5644
  2014-08-31 21:16:32,506 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., current region memstore size 195.8k
  2014-08-31 21:16:38,151 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting 
  2014-08-31 21:16:38,156 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a with permission=rwxrwxrwx
  2014-08-31 21:16:38,157 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 21:16:38,157 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 21:16:38,165 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 21:16:38,166 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a: CompoundBloomFilterWriter
  2014-08-31 21:16:38,188 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a)
  2014-08-31 21:16:38,189 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=3316344, memsize=180.3k, into tmp file hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a
  2014-08-31 21:16:38,197 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/67548981c14c48abbdd66208436e150a to hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
  2014-08-31 21:16:38,208 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, entries=580, sequenceid=3316344, filesize=106.7k
  2014-08-31 21:16:38,210 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~195.8k/200488, currentsize=0.0/0 for region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. in 59ms, sequenceid=3316344, compaction requested=true
  2014-08-31 21:16:38,210 DEBUG org.apache.hadoop.hbase.regionserver.Store: d364a5f6bc7e54e72d915a3198636afd - a: Initiating minorcompaction
  2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on a in region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
  2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into tmpdir=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp, seqid=3316344, totalSize=4.0m
  2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on a in region messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd.
  2014-08-31 21:16:38,222 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into tmpdir=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp, seqid=3316344, totalSize=4.0m
  2014-08-31 21:16:38,222 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/6f8d2fc1160245318ddec3f193982d9c, keycount=7820, bloomtype=NONE, size=3.6m, encoding=NONE
  2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, keycount=592, bloomtype=NONE, size=250.5k, encoding=NONE
  2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: Compacting hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, keycount=580, bloomtype=NONE, size=106.7k, encoding=NONE
  2014-08-31 21:16:38,223 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction requested: regionName=messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., storeName=a, fileCount=3, fileSize=4.0m (3.6m, 250.5k, 106.7k), priority=3, time=7733965975674385; Because: regionserver60020.cacheFlusher; compaction_queue=(0:0), split_queue=0
  2014-08-31 21:16:38,224 DEBUG org.apache.hadoop.hbase.util.FSUtils: Creating file=hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21 with permission=rwxrwxrwx
  2014-08-31 21:16:38,225 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultReplication
  2014-08-31 21:16:38,225 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support getDefaultBlockSize
  2014-08-31 21:16:38,229 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
  2014-08-31 21:16:38,229 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21: CompoundBloomFilterWriter
  2014-08-31 21:16:38,383 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21)
  2014-08-31 21:16:38,396 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/.tmp/0cfab02d23184841b0185dbcb3c7cb21 to hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/0cfab02d23184841b0185dbcb3c7cb21
  2014-08-31 21:16:38,406 DEBUG org.apache.hadoop.hbase.regionserver.Store: Removing store files after compaction...
  2014-08-31 21:16:38,409 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving compacted store files.
  2014-08-31 21:16:38,410 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Starting to archive files:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, ..........................................
  for:hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, free to archive original file.
  2014-08-31 21:16:38,419 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b, to: hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/75cb807d466940d98acc15ad295d374b
  2014-08-31 21:16:38,419 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
  2014-08-31 21:16:38,420 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: No existing file in archive for:hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, free to archive original file.
  2014-08-31 21:16:38,423 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://hadoop:54310/hbase/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a, to: hdfs://hadoop:54310/hbase/.archive/messages/d364a5f6bc7e54e72d915a3198636afd/a/67548981c14c48abbdd66208436e150a
  2014-08-31 21:16:38,423 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction of 3 file(s) in a of messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd. into 0cfab02d23184841b0185dbcb3c7cb21, size=4.0m; total size for store is 332.2m
  2014-08-31 21:16:38,423 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=messages,05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.d364a5f6bc7e54e72d915a3198636afd., storeName=a, fileCount=3, fileSize=4.0m, priority=3, time=7733965975674385; duration=0sec
  2014-08-31 21:16:38,424 DEBUG org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: CompactSplitThread status: compaction_queue=(0:0), split_queue=0
  2014-08-31 21:16:40,082 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for messages,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.5d9b973aef7e10eee0e32fecdbeb37e9., current region memstore size 245.3k
  2014-08-31 21:16:40,083 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting messages,03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1402123407977.5d9b973aef7e10eee0e32fecdbeb37e9., syncing WAL and waiting on mvcc, flushsize=251224
  2014-08-31 21:16:40,083 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
  </code></pre>