title: Why is a 'for' loop faster than a 'while' loop for reading from a file?
body: <h3>Preface:</h3>
  <p>I've always been taught that when working in the shell, it is preferable to do <code>while</code> loops over <code>for</code> loops, and that you should not use <code>for</code> loops with command substitution <code>cat</code>'ing a file. My understanding was that there are a number of reasons for this, including:</p>
  <ul>
  <li>A <code>for</code> loop requires loading all of the data to be processed into memory all at once</li>
  <li><code>for</code> loops do word splitting on whitespace by default instead of newlines, so in addition to having to have all of your in file in memory, you also have more word splitting going on taking up memory</li>
  <li><code>for</code> loops don't begin processing &quot;to the right of the do&quot; until everything in your <code>in</code> statement is done loading, which means that part of the time you're waiting for results, nothing is actually happening while you're &quot;pre-loading&quot;.</li>
  </ul>
  <p>However, in doing some simple tests, I've found that while memory consumption seems greater in <code>for</code> loops (as is expected) the actual performance of <code>while</code> loops is lower. It's not a huge difference, and at the scale that this would likely start to matter on any modern machine, I'd probably be switching over to awk or python, but I'm still curious why this is happening.</p>
  <h3>Test setup:</h3>
  <p>I did a simple series of tests of just echoing lines of a file into /dev/null. My input is two flat files containing 100K and 1Mil IP addresses respectively. In my output below is one test, but I ran this several times with similar results each time. I was running this test on a 2013 MBA (i7, 8g Mem).</p>
  <h3>Test results</h3>
  <pre><code>Ds-MacBook-Air:~ d$ time for i in $(cat /tmp/ips.100k);do echo $i &gt; /dev/null;done
  
  real    0m1.629s
  user    0m1.154s
  sys 0m0.480s
  Ds-MacBook-Air:~ d$ time for i in $(cat /tmp/ips.mill);do echo $i &gt; /dev/null;done
  
  real    0m17.567s
  user    0m12.414s
  sys 0m5.131s
  Ds-MacBook-Air:~ d$ time while read i;do echo $i &gt; /dev/null;done &lt; /tmp/ips.100k
  
  real    0m2.148s
  user    0m1.493s
  sys 0m0.655s
  Ds-MacBook-Air:~ d$ time while read i;do echo $i &gt; /dev/null;done &lt; /tmp/ips.mill
  
  real    0m21.536s
  user    0m14.915s
  sys 0m6.617s
  
  Ds-MacBook-Air:~ d$ tail -5 /tmp/ips.100k /tmp/ips.mill
  ==&gt; /tmp/ips.100k &lt;==
  1.1.134.155
  1.1.134.156
  1.1.134.157
  1.1.134.158
  1.1.134.159
  
  ==&gt; /tmp/ips.mill &lt;==
  1.15.66.59
  1.15.66.60
  1.15.66.61
  1.15.66.62
  1.15.66.63
  
  Ds-MacBook-Air:~ d$ wc -l /tmp/ips.100k /tmp/ips.mill
    100000 /tmp/ips.100k
   1000000 /tmp/ips.mill
   1100000 total
  </code></pre>
  <hr />
  <p>I don't have any direct citations on what I've asserted regarding <code>for</code> loops vs <code>while</code> loops, but I specifically be it covered in either the ~~TLDP~~ Wooldridge documentation, or another Bash programming guide (some quick googling doesn't yield the exact place where I read most of this years ago.)</p>