title: Hadoop - appropriate block size for unsplittable files of varying sizes (200-500mb)
body: <p>If I need to do a sequential scan of (non-splittable) thousands of gzip files of sizes varying between 200 and 500mb, what is an appropriate block size for these files?</p>
  
  <p>For the sake of this question, let's say that the processing done is very fast, so restarting a mapper is not costly, even for large block sizes.</p>
  
  <p>My understanding is:</p>
  
  <ol>
  <li>There's hardly an upper limit of block size, as there's "plenty of files" for an appropriate amount of mappers for the size of my cluster.    </li>
  <li>To ensure data-locality, I want each gzip file to be in 1 block.</li>
  </ol>
  
  <p>However, the gzipped files are of varying sizes. How is data stored if I choose a block size of ~500mb (e.g. max file size of all my input files)? Would it be better to pick a "very large" block size, like 2GB? Is HDD capacity wasted excessively in either scenario?</p>
  
  <p>I guess I'm really asking how files are actually stored and split across hdfs blocks - as well as trying to gain an understanding of best practice for non-splittable files.</p>
  
  <p><strong>Update: A concrete example</strong></p>
  
  <p>Say I'm running a MR Job on three 200 MB files, stored as in the following illustration.</p>
  
  <p>If HDFS stores files as in case A, 3 mappers would be guaranteed to be able to process a "local" file each. However, if the files are stored as in case B, one mapper would need to fetch part of file 2 from another data node. </p>
  
  <p>Given there's plenty of free blocks, does HDFS store files as illustrated by case A or case B?</p>
  
  <p><img src="https://i.stack.imgur.com/7MMFY.png" alt="HDFS strategies"></p>