title: Fastest way to index large sorted hash file
body: <p>I am building a file-based index for the sorted <a href="https://haveibeenpwned.com/Passwords" rel="nofollow noreferrer">haveibeenpwned</a> passwords text file and it got me wondering what's the fastest way to do this?    </p>
  
  <p>I figured a good way to build a quickly grep-able index would be split the sorted file into 256 files named with the first two hex digits (i.e. FF.txt, FE.txt, etc).  I found ripgrep <code>rg</code> to be about 5 times faster than grep on my computer.  So I tried something like this:</p>
  
  <pre><code>for i in {255..0} 
  do
      start=$(date +%s)
      hex="$(printf  '%02x' $i  | tr [:lower:] [:upper:])"
      rg "^$hex" pwned-passwords-ntlm-ordered-by-hash-v4.txt &gt; ntlm/$hex-ntlm.txt
      echo 0x$hex completed in $(($(date +%s) - $start)) seconds
  done
  </code></pre>
  
  <p>This is the fastest solution I could come up with.  ripgrep is able to create each file in 25 seconds.  So I'm looking at about 100 minutes to create this index.  When I split the job in half, and run them in parallel, each pair of files is created in 80 seconds.   So it seems best to just let ripgrep work its magic and work in series.</p>
  
  <p>Obviously, I won't be indexing this list too often, but it's just fun to think about.  Any thoughts on a faster way (aside from using a database) to index this file?</p>