performance problem
<p>OK so I need to know if anyone can see a way to reduce the number of iterations of these loops because I can't. The first while loop is for going through a file, reading one line at a time. The first foreach loop is then comparing each of the compareSet with what was read in the first while loop. Then the next while loop is to do with bit counting.</p>

<p>As requested, an explaination of my algorithm:</p>

<p>There is a file that is too large to fit in memory. It contains a word followed by the pages in a very large document that this word is on. EG:</p>

<p>sky 1 7 9 32....... (it is not in this format, but you get the idea).</p>

<p>so parseLine reads in the line and converts it into a list of ints that are like a bit array where 1 means the word is on the page, and 0 means it isn't.</p>

<p>CompareSet is a bunch of other words. I can't fit my entire list of words into memory so I can only fit a subset of them. This is a bunch of words just like the "sky" example. I then compare each word in compareSet with Sky by seeing if they are on the same page.</p>

<p>So if sky and some other word both have 1 set at a certain index in the bit array (simulated as an int array for performance), they are on the same page. The algorithm therefore counts the occurances of any two words on a particular page. So in the end I will have a list like:</p>

<p>(for all words in list) is on the same page as (for all words in list) x number of times.</p>

<p>eg sky and land is on the same page x number of times.</p>

<pre><code> while ((line = parseLine(s)) != null) {
     getPageList(line.Item2, compareWord);
     foreach (Tuple&lt;int, uint[], List&lt;Tuple&lt;int, int&gt;&gt;&gt; word in compareSet) {
         unchecked {
             for (int i = 0; i &lt; 327395; i++) {
                 if (word.Item2[i] == 0 || compareWord[i] == 0)
                     continue;
                 uint combinedNumber = word.Item2[i] &amp; compareWord[i];
                 while (combinedNumber != 0) {
                     actual++;
                     combinedNumber = combinedNumber &amp; (combinedNumber - 1);
                 }
             }
         }
</code></pre>
