title: Not able to perform wide HBase table scans
body: <p>I'm facing a problem due to a BAD table design in HBase. The problem is millions of records end up under the same row key (1 cf). Till 2.5M I was able to run mapReduces with Spark by scanning a single row, but now some of the rows are reaching 5 or 6 millions and whenever I perform a scan or get, all my regionservers are down within a couple of minutes. I'm working with HDP 2.2 and HBase 0.98.4.2.2</p>
  
  <p>So far I've tried:</p>
  
  <pre><code>confPoints.setInt("hbase.rpc.timeout",6000000)
  ...
  scanPoints.setBatch(1000)
  </code></pre>
  
  <p>Before creating a new table with new rowkey design I really need to process this data. I'm new to HBase so maybe some of this suggestions sound stupid but:</p>
  
  <ul>
  <li>Would increasing java heap size help in any way? </li>
  <li>Is there a possibility of splitting a row into 2 or more rows? </li>
  <li>Can I run a MapReduce over the raw stored data in HDFS without passing through HBase? </li>
  <li>Any other idea?</li>
  </ul>
  
  <p>Thanks!</p>
  
  <p><strong>EDITED:</strong></p>
  
  <p>Actually, the second option I think it is not feasible cause hbase doesn't let to update records, just delete + create again.</p>
  
  <p><strong>EDITED 2:</strong></p>
  
  <p>Each record in a row is about tens of bytes. The problem when having millions of records per row is when trying to scan this kind of row is that after a couple of minutes region servers start to go down one by one. Maybe trying to get a row of 512MB aprox. is too big for my cluster configuration: 6 nodes of 8GB each.</p>
  
  <p>Searching in the HBase logs the only exception I can find is this on:</p>
  
  <pre><code>2015-08-25 15:07:19,722 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-0] handler.OpenRegionHandler: Opened my-hbase-table,20150807.33,1439222912086.e731d603bb5d1f0d593736eab922069c. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:19,724 INFO  [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Replaying edits from hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
  2015-08-25 15:07:19,725 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioned node 4945982779c1cba7b1726e77a45d405a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  2015-08-25 15:07:19,725 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] handler.OpenRegionHandler: Transitioned 4945982779c1cba7b1726e77a45d405a to OPENED in zk on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:19,726 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] handler.OpenRegionHandler: Opened my-hbase-table,20150727.33,1438203991635.4945982779c1cba7b1726e77a45d405a. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:19,733 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Attempting to retransition opening state of node 3bc481ff534f0907e6b99d5eff1793f5
  2015-08-25 15:07:19,734 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Applied 0, skipped 1, firstSequenceidInLog=11099011, maxSequenceidInLog=11099011, path=hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
  2015-08-25 15:07:19,734 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Empty memstore size for the current region my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
  2015-08-25 15:07:19,737 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Deleted recovered.edits file=hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
  2015-08-25 15:07:19,759 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] wal.HLogUtil: Written region seqId to file:hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/11099013_seqid ,newSeqId=11099013 ,maxSeqId=11099010
  2015-08-25 15:07:19,761 INFO  [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Onlined 3bc481ff534f0907e6b99d5eff1793f5; next sequenceid=11099013
  2015-08-25 15:07:19,764 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Attempting to retransition opening state of node 3bc481ff534f0907e6b99d5eff1793f5
  2015-08-25 15:07:19,773 INFO  [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.HRegionServer: Post open deploy tasks for region=my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
  2015-08-25 15:07:19,773 DEBUG [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.CompactSplitThread: Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0, merge_queue=0
  2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store files, 0 compacting, 4 eligible, 10 blocking
  2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio
  2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate.
  2015-08-25 15:07:19,775 DEBUG [regionserver60020-smallCompactions-1440529300855] regionserver.CompactSplitThread: Not compacting my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. because compaction request was cancelled
  2015-08-25 15:07:19,787 INFO  [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] catalog.MetaEditor: Updated row my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. with server=ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:19,787 INFO  [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.HRegionServer: Finished post open deploy task for my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
  2015-08-25 15:07:19,788 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioning 3bc481ff534f0907e6b99d5eff1793f5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioned node 3bc481ff534f0907e6b99d5eff1793f5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
  2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] handler.OpenRegionHandler: Transitioned 3bc481ff534f0907e6b99d5eff1793f5 to OPENED in zk on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] handler.OpenRegionHandler: Opened my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
  2015-08-25 15:07:20,344 INFO  [B.DefaultRpcServer.handler=3,queue=3,port=60020] regionserver.HRegionServer: Client tried to access missing scanner 1
  2015-08-25 15:07:20,346 DEBUG [B.DefaultRpcServer.handler=3,queue=3,port=60020] ipc.RpcServer: B.DefaultRpcServer.handler=3,queue=3,port=60020: callId: 36 service: ClientService methodName: Scan size: 25 connection: 172.31.40.100:42285
  org.apache.hadoop.hbase.UnknownScannerException: Name: 1, already closed?
  at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3150)
  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
  at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p><strong>EDITED 3:</strong></p>
  
  <p>I've tried to do range scans within a row with a ColumnRangeFilter and it is working without putting down any region servers:</p>
  
  <pre><code>scan 'my-table', {STARTROW=&gt;'row-key',ENDROW=&gt;'row-key', FILTER=&gt; ColumnRangeFilter.new(Bytes.toBytes('first_possible_column_prefix'),true,Bytes.toBytes('another_possible_column_prefix’),false)}
  </code></pre>
  
  <p>This code in Spark though is putting the region servers down, same behavior as before:</p>
  
  <pre><code>val scanPoints = new Scan()
  scanPoints.setStartRow((queryDate+"."+venueId).getBytes())
  scanPoints.setStopRow((queryDate+"."+venueId+"1").getBytes())
  scanPoints.setFilter(new ColumnRangeFilter(Bytes.toBytes("first_possible_column_prefix"),true,Bytes.toBytes("another_possible_column_prefix"),false))
  ...
  val confPoints = HBaseConfiguration.create()
  confPoints.set(TableInputFormat.INPUT_TABLE, Utils.settings.HBaseWifiVisitorsTableName)
  confPoints.set("hbase.zookeeper.quorum", Utils.settings.zQuorum);
  confPoints.setInt("zookeeper.session.timeout", 6000000)
  confPoints.set("hbase.zookeeper.property.clientPort", Utils.settings.zPort);
  confPoints.set("zookeeper.znode.parent",Utils.settings.HBaseZNode)
  confPoints.set("hbase.master", Utils.settings.HBaseMaster)
  confPoints.set("hbase.mapreduce.scan.column.family","positions")
  confPoints.setLong("hbase.client.scanner.max.result.size",2147483648L)
  confPoints.setLong("hbase.server.scanner.max.result.size",2147483648L)
  confPoints.setInt("hbase.rpc.timeout",6000000)
  confPoints.setInt("hbase.client.operation.timeout",6000000)
  confPoints.set(TableInputFormat.SCAN, convertScanToString(scanPoints))
  ...
  val rdd = sc.newAPIHadoopRDD(confPoints, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result]).cache()
  </code></pre>
  
  <p>If I would be able to make this Spark Job to work, I could iterate through the whole row scanning intervals to process it entirely.</p>