title: Estimating size of a large text file
body: <p>From the comment on a question about selecting <em>n</em> random lines from a text file here:</p>
  
  <p><a href="https://stackoverflow.com/questions/9245638/select-random-lines-from-a-file#comment57481367_15065490">Select random lines from a file</a></p>
  
  <p>User commented that they used the <strong>shuf</strong> command to randomly select lines from a text file having 78 billion lines in less than a minute.</p>
  
  <p>I see from various sources on the internet that people have text file sizes varying from 100GB-200GB-300GB for mere 7-15 Billion lines, depending on the metadata. </p>
  
  <p><strong>Assuming we have:</strong></p>
  
  <ol>
  <li><p>Text file containing ASCII characters, where we define a line break for newline after each 100 characters. This file has 78 Billion lines.</p></li>
  <li><p>We have system with compute capacity as :</p>
  
  <p>a. RAM - 16GB</p>
  
  <p>b. Processor - 2.5 GHz Intel Core i7</p>
  
  <p>c. Disk - 500GB SSD</p></li>
  </ol>
  
  <p><strong>I am curious as to :</strong></p>
  
  <ol>
  <li>What will be the estimate size of the text file? </li>
  </ol>
  
  <p>Will it also depend on how different OSs encode and store characters at memory level? If yes, then how much will it factor into the size calculation?</p>
  
  <ol start="2">
  <li><p>Ideally how much time <strong>bash</strong> - running on the system (with above mentioned specifications) - will take to process this text file with <strong>shuf</strong> command?</p></li>
  <li><p>If the text file size comes in TBs, then how to feed data to the system? 
  How will <strong>shuf</strong> operate for such large files with maximum efficiency in the mentioned system.</p></li>
  </ol>
  
  <p><strong>shuf</strong> performance in my system(specs above):</p>
  
  <ul>
  <li><p>100 characters per line : 1 line</p>
  
  <p> FILE SIZE : ~ 100Bytes </p>
  
  <p>    TIME : <em>real</em>   0m0.025s   <em>user</em>   0m0.007s    <em>sys</em>   0m0.013s</p></li>
  <li><p>100 characters per line : 100,000 lines   </p>
  
  <p>FILE SIZE : ~10MB  </p>
  
  <p>TIME :  <em>real</em>   0m0.122s   <em>user</em>   0m0.036s    <em>sys</em>   0m0.080s</p></li>
  <li><p>100 characters per line : 100,000,000 lines   </p>
  
  <p>FILE SIZE : ~10GB   </p>
  
  <p>TIME : <em>real</em>    9m37.108s  <em>user</em>   2m22.011s  <em>sys</em>    3m3.659s</p></li>
  </ul>
  
  <p>[nit : For those who are interested, here are some reddit meme threads for the same :
  <a href="https://www.reddit.com/r/ProgrammerHumor/comments/grsreg/shuf_go_brrr/" rel="nofollow noreferrer">https://www.reddit.com/r/ProgrammerHumor/comments/grsreg/shuf_go_brrr/</a><br>
  <a href="https://www.reddit.com/r/ProgrammerHumor/comments/groe6y/78_billion_prayers/" rel="nofollow noreferrer">https://www.reddit.com/r/ProgrammerHumor/comments/groe6y/78_billion_prayers/</a> ]</p>
  
  <p>Edit#1:
  Refining question to have more detail. Adding more resources as per comments and findings.</p>
  
  <p>Edit#2:
  Added shuf performance in my system for different text file sizes</p>