W-shingling implementation - storing shingles
<p>I need to implement <a href="http://en.wikipedia.org/wiki/W-shingling" rel="nofollow">w-shingling</a> (in Java) to compare similirarity of two html documents. The question is how to gather and store shingles. Let's assume that (a,rose,is,a,rose,is,a,rose) is one of these documents. I guess that my alorithm (with LinkedList) won't be the fastest:</p>

<ol>
<li>Get next single word from document (a) - if there are no other words, stop here.</li>
<li>Check (a) occurency in shingles list</li>
<li>If it occurs there, go to first step</li>
<li>If it isn't, attach it to the list and go to the first step</li>
</ol>

<p>As I predict, this could be extremeally slow with large documents. Can you give me some hints to make it faster...? </p>
