title: Why does scan for large blob value crash HBase cluster?
body: <p>I have an HBase table in which I have two column families, 'i:*' for info, and 'f:b' for file:blob.
  I am storing images in the value, some images are almost 12MB.</p>
  
  <p>I can load/insert the files no problem in java, but as soon as I try to retrieve them via a scan for the f:b family value (the blobs), my scanner sits until it times out and every region server on my cluster dies in sequence (I have a 20 node cluster). The only way to stop this quasi virus that my scanner somehow inflicts upon my helpless nodes is to drop the table altogether (or so it seems).</p>
  
  <p>I am using Cloudera EDH '0.98.6-cdh5.2.0'</p>
  
  <p>unfortunately my client just times out, so no valuable exceptions there, and all I could get from my node logs are below</p>
  
  <pre><code>2014-10-27 21:47:36,106 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to archive class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp
  java.io.FileNotFoundException: File hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp does not exist.
         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:658)
          at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:104)
   at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:716)
     at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
       at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:712)
     at org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath.getChildren(HFileArchiver.java:628)
        at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:346)
          at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:347)
    at org.apache.hadoop.hbase.backup.HFileArchiver.resolveAndArchive(HFileArchiver.java:284)
      at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:137)
    at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:75)
   at org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.java:333)
         at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:254)
      at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:101)
          at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
          at java.lang.Thread.run(Thread.java:745)
  2014-10-27 21:47:36,129 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to complete archive of: [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits/0000000000000000029.temp]. Those files are still in the original location, and they may slow down reads.
  2014-10-27 21:47:36,129 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog table
  java.io.IOException: Received error when attempting to archive files ([class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/f, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/i, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://nameservice1/hbase/data/default/RASTER/92ceb2d86662ad6d959f4cc384229e0f/recovered.edits]), cannot delete region directory. 
          at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:148)
        at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:75)
       at org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.java:333)
     at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:254)
          at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:101)
      at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
      at java.lang.Thread.run(Thread.java:745)
  2014-10-27 21:47:36,146 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitWAL/WALs%2Finsight-staging-slave019.spadac.com%2C60020%2C1414446135179-splitting%2Finsight-staging-slave019.spadac.com%252C60020%252C1414446135179.1414446317771
  </code></pre>
  
  <p>here is my code for scanning the table</p>
  
  <pre><code>    try {
        if (hBaseConfig == null) {
          hBaseConfig = HBaseConfiguration.create();
          hBaseConfig.setInt("hbase.client.scanner.timeout.period", 1200000);
          hBaseConfig.set("hbase.client.keyvalue.maxsize", "0");
          hBaseConfig.set("hbase.master", PROPS.get().getProperty("hbase.master"));
          hBaseConfig.set("hbase.zookeeper.quorum", PROPS.get().getProperty("zks"));
          hBaseConfig.set("zks.port", "2181");
          table = new HTable(hBaseConfig, "RASTER");
        }
  
        Scan scan = new Scan();
        scan.addColumn("f".getBytes(), "b".getBytes());
        scan.addColumn("i".getBytes(), "name".getBytes());
        ResultScanner scanner = table.getScanner(scan);
  
        for (Result rr = scanner.next(); rr != null; rr = scanner.next()) {
  /*I NEVER EVEN GET HERE IF I SCAN FOR 'f:b'*/
          CellScanner cs = rr.cellScanner();
          String name = "";
          byte[] fileBs = null;
          while (cs.advance()) {
  
            Cell current = cs.current();
  
            byte[] cloneValue = CellUtil.cloneValue(current);
            byte[] cloneFamily = CellUtil.cloneFamily(current);
            byte[] qualBytes = CellUtil.cloneQualifier(current);
            String fam = Bytes.toString(cloneFamily);
            String qual = Bytes.toString(qualBytes);
            if (fam.equals("i")) {
  
              if (qual.equals("name")) {
                name = Bytes.toString(cloneValue);
              }
            } else if (fam.equals("f") &amp;&amp; qual.equals("b")) {
              fileBs = cloneValue;
            }
  
          }
  
          OutputStream bof = new FileOutputStream("c:\\temp\\" + name);
          bof.write(fileBs);
          break;
  
        }
      } catch (IOException ex) {
        //removed
      }
  </code></pre>
  
  <p>thanks
  Does anyone know why a scan for a large blob might annihilate my cluster? I am sure it's silly just can't figure it out.</p>