title: fast loading of large hash table in Perl
body: <p>I have about 30 text files with the structure</p>
  
  <pre><code>wordleft1|wordright1
  wordleft2|wordright2
  wordleft3|wordright3
  ...
  </code></pre>
  
  <p>The total size of the files is about 1 GB with about 32 million lines of word combinations. </p>
  
  <p>I tried a few approaches to load them as fast as possible and store the combinations within a hash</p>
  
  <pre><code>$hash{$wordleft} = $wordright
  </code></pre>
  
  <p>Opening file by file and reading line by line takes about 42 seconds. I then store the hash with the Storable module </p>
  
  <pre><code>store \%hash, $filename
  </code></pre>
  
  <p>Loading the data again</p>
  
  <pre><code>$hashref = retrieve $filename
  </code></pre>
  
  <p>reduces the time to about 28 seconds. I use a fast SSD drive and a fast CPU and have enough RAM to hold all the data (it takes about 7 GB).</p>
  
  <p>I'm searching for a faster way to load this data into the RAM (I can't keep it there for a few reasons).</p>