title: Finding longest common sequences in big data
body: <p>I have logs from a bunch (millions) of small experiments.</p>
  
  <p>Each log contains a list (tens to hundreds) of entries. Each entry is a timestamp and an event ID (there are several thousands of unique event IDs, each of them may occur several times in the logs). Here is an example of the log of one experiment:</p>
  
  <pre>
  1403973044 alpha
  1403973045 beta
  1403973070 gamma
  1403973070 alpha
  1403973098 delta
  </pre>
  
  <p>I need to find sequences that are common for a number of experiments.</p>
  
  <p>A sequence is several (at least two) event IDs that follow each other:</p>
  
  <pre>
  alpha
  beta
  gamma
  </pre>
  
  <p>Common sequence is a sequence that can be found in at least two logs. Logs</p>
  
  <pre>
  alpha
  beta
  gamma
  delta
  epsilon
  </pre>
  
  <p>and</p>
  
  <pre>
  iota
  beta
  gamma
  epsilon
  kappa
  </pre>
  
  <p>have a common sequence:</p>
  
  <pre>
  beta
  gamma
  epsilon
  </pre>
  
  <p>(Obviously, I'm looking for longest common sequence here.)</p>
  
  <p>I am interested in large sequences that occur often. I do not know cutoff values beforehand. Say, I need top 10 (or, maybe, top 100 sequences) by length among top 1000 (or top 30%) by occurrence. (This criteria is given at random and can be changed almost arbitrarily to ease data processing.)</p>
  
  <p>Any advice on how I do this in a scalable way?</p>
  
  <p>I would prefer to minimize the amount of code that I have, but I would not like to use proprietary programs to get the solution.</p>
  
  <p>...I think that a somewhat similar problem is solved in bioinformatics. <s>But they have much shorter alphabet than me :-)</s> Wrong! Proteomics deals with much larger alphabets! (Thanks to @JayInNyc)</p>