title: GNU Parallel: split file into children
body: <p><strong>Goal</strong></p>
  
  <p>Use GNU Parallel to split a large .gz file into children. Since the server has 16 CPUs, create 16 children. Each child should contain, at most, N lines. Here, N = 104,214,420 lines. Children should be in .gz format.</p>
  
  <p><strong>Input File</strong></p>
  
  <ul>
  <li>name: file1.fastq.gz</li>
  <li>size: 39 GB</li>
  <li>line count: 1,667,430,708 (uncompressed)</li>
  </ul>
  
  <p><strong>Hardware</strong></p>
  
  <ul>
  <li>36 GB Memory</li>
  <li>16 CPUs</li>
  <li>HPCC environment (I'm not admin)</li>
  </ul>
  
  <p><strong>Code</strong></p>
  
  <p><em>Version 1</em></p>
  
  <pre><code>zcat "${input_file}" | parallel --pipe -N 104214420 --joblog split_log.txt --resume-failed "gzip &gt; ${input_file}_child_{#}.gz"
  </code></pre>
  
  <p>Three days later, the job was not finished. split_log.txt was empty. No children were visible in the output directory. Log files indicated that Parallel had increased the <code>--block-size</code> from 1 MB (the default) to over 2 GB. This inspired me to change my code to Version 2.</p>
  
  <p><em>Version 2</em></p>
  
  <pre><code># --block-size 3000000000 means a single record could be 3 GB long. Parallel will increase this value if needed.
  
  zcat "${input_file}" | "${parallel}" --pipe -N 104214420 --block-size 3000000000 --joblog split_log.txt --resume-failed "gzip &gt; ${input_file}_child_{#}.gz"
  </code></pre>
  
  <p>The job has been running for ~2 hours. split_log.txt is empty. No children are visible in the output directory yet. So far, log files show the following warning:</p>
  
  <pre><code>parallel: Warning: --blocksize &gt;= 2G causes problems. Using 2G-1.
  </code></pre>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li>How can my code be improved ?</li>
  <li>Is there a faster way to accomplish this goal ?</li>
  </ol>