title: storing images in HBASE for processing and quick access
body: <p>I have a large number of image files that I need to store and process on HDFS</p>
  
  <p>Let's assume 2 scenarios:</p>
  
  <ol>
  <li>Images are less than 5MB</li>
  <li>Images range from 50KB to 20MB</li>
  </ol>
  
  <p>I would like to do 4 things with the images:</p>
  
  <ol>
  <li>I need to apply some function fnc() to each image independently.</li>
  <li>I will need to extract a specific image from HDFS from time to time (1000 times/day) and display it on a website. <strong>These are user queries for specific images so the latency should be a few seconds</strong>.</li>
  <li>Once a year groups of images would have to be deleted.</li>
  <li><strong>New images will be added to the system (1000 new images/day)</strong></li>
  </ol>
  
  <p>The solution design IMO should consider:</p>
  
  <ol>
  <li>The small files issue: </li>
  <li>MR Processing</li>
  <li>Quick access to the files</li>
  <li>Quick write of the new files is not that big of an issue since the image will not be used immediately. A delay of a few minutes or hours is OK.</li>
  </ol>
  
  <p>My first thought was to aggregate the images to take care of the small file issue, which satisfied 1 and 2. <strong>But I was left with the quick random access to the images problem and with the addition of new images. I am not sure how I could deal with this</strong>.</p>
  
  <p>I looked into sequenceFiles, HAR, mapFiles, combineFileInputFormat, Avro, but wasn't able to find a solution for (3) and (4). Since I would have to take care of indexing the contents of the blocks and searching and deleting or adding new files may become tricky.</p>
  
  <p>The other approach was to use HBase or HCatalog to store the images, this would take care of (1) (2) (3) and (4), but at what cost? I know that storing binary BLOBS in a database is not very efficient specially as the number of images increases but I thought maybe HBase or HCatalog handled this a bit different. </p>
  
  <p>Thanks for all the help!</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>I just found this thread on HBase for serving images, apparently Yfrog and ImageShack have billions of records with images here is the <a href="http://apache-hbase.679495.n3.nabble.com/Storing-images-in-Hbase-td4036184.html" rel="noreferrer">link</a> it's a good read. Although If anyone knows of any benchmarks that would be great.</p>