title: Count subsequences in hundreds of GB of data
body: <p>I'm trying to process a very large file and tally the frequency of all sequences of a certain length in the file. </p>
  
  <p>To illustrate what I'm doing, consider a small input file containing the sequence <code>abcdefabcgbacbdebdbbcaebfebfebfeb</code></p>
  
  <p>Below, the code reads the whole file in, and takes the first substring of length n (below I set this to 5, although I want to be able to change this) and counts its frequency:  </p>
  
  <pre><code>abcde =&gt; 1
  </code></pre>
  
  <p>Next line, it moves one character to the right and does the same: </p>
  
  <pre><code>bcdef =&gt; 1
  </code></pre>
  
  <p>It then continues for the rest of the string and prints the 5 most frequent sequences:</p>
  
  <pre><code>open my $in, '&lt;', 'in.txt' or die $!; # 'abcdefabcgbacbdebdbbcaebfebfebfeb'
  
  my $seq = &lt;$in&gt;; # read whole file into string
  my $len = length($seq);
  
  my $seq_length = 5; # set k-mer length
  my %data;
  
  for (my $i = 0; $i &lt;= $len - $seq_length; $i++) {
       my $kmer = substr($seq, $i, $seq_length);
       $data{$kmer}++;
  }
  
  # print the hash, showing only the 5 most frequent k-mers
  my $count = 0;
  foreach my $kmer (sort { $data{$b} &lt;=&gt; $data{$a} } keys %data ){
      print "$kmer $data{$kmer}\n";
      $count++;
      last if $count &gt;= 5;
  }
  </code></pre>
  
  <hr>
  
  <pre><code>ebfeb 3
  febfe 2
  bfebf 2
  bcaeb 1
  abcgb 1
  </code></pre>
  
  <hr>
  
  <p>However, I would like to find a more efficient way of achieving this. If the input file was 10GB or 1000GB, then reading the whole thing into a string would be very memory expensive. </p>
  
  <p>I thought about reading in blocks of characters, say 100 at a time and proceeding as above, but here, sequences that span 2 blocks would not be tallied correctly. </p>
  
  <p>My idea then, is to only read in n number of characters from the string, and then move onto the next n number of characters and do the same, tallying their frequency in a hash as above. </p>
  
  <ul>
  <li>Are there any suggestions about how I could do this? I've had a look a <a href="http://perldoc.perl.org/functions/read.html" rel="nofollow">read</a> using an offset, but can't get my head around how I could incorporate this here</li>
  <li>Is <code>substr</code> the most memory efficient tool for this task? </li>
  </ul>