title: Large-scale processing of seralized Integer objects
body: <p>I have a large data set in the following format:</p>
  
  <p>In total, there are 3687 object files. Each of which contains 2,000,000 records. Each file is 42MB in size.</p>
  
  <p>Each record contains the following:</p>
  
  <ul>
  <li>An id  (Integer value)</li>
  <li>Value1 (Integer)</li>
  <li>Value2 (Integer)</li>
  <li>Value3 (Integer)</li>
  </ul>
  
  <p>The content of each file is not sorted or ordered in any way as they are observed during a data collection process.</p>
  
  <p>Ideally, I want to build an index for this data. (Indexed by the id) which would mean the following:</p>
  
  <ol>
  <li><p>Dividing the set of ids into manageable chunks. </p></li>
  <li><p>Scanning the files to get data related to the current working set of ids.</p></li>
  <li><p>Build the index.</p></li>
  <li><p>Go over the next chunk and repeat 1,2,3.</p></li>
  </ol>
  
  <p>To me this sounds fine but loading 152GB back and forth is time-consuming and wonder about the best possible approach or even whether Java is actually the right language to use for such a process.</p>
  
  <p>I've 256GB of ram and 32 cores on my machine.</p>
  
  <hr>
  
  <p><strong>Update:</strong>
  Let me modify this, putting aside I/O, and assuming the file is in-memory in a byte array. </p>
  
  <p>What would be the fastest possible way to decode a 42MB Object file that have 2,000,000 records and each record contains 4 Integers serialized. </p>