title: Working with very huge XML file in C#
body: <p>I have this very huge XML file of size 2.8GB. This is Polish Wikipedia's articles dump. The size of this file is very problematic for me. The task is to <strong>search</strong> this file for some big amount of data. All I have are titles of the articles. I thought that I could sort that titles and use one linear loop through the file. Idea is not so bad, but articles are <strong>not</strong> sorted alphabetically. They are sorted by ID, which I don't know <em>a priori</em>.</p>
  
  <p>So, my second thought was to make an index of that file. To store in other file (or database) lines in following format: <code>title;id;index</code> (maybe without an ID). I my other question I asked for help with that. The hypothesis was that if I had index of needed tag I could use just simple <code>Seek</code> method to move the cursor within the file without reading all content, etc. For smaller files I think this could work fine. But on my computer (laptop, C2D proc, Win7, VS2008) I get error that application is not responding. </p>
  
  <p>In my program, I am reading each line from file and checks if it contains a tag that I need. I am also counting all bytes I read and save lines in format mentioned above. So while indexing program gets hung up. But till then the result index file is 36.2MB and the last index is like 2,872,765,202 (B) while whole XML file is 3,085,439,630 B long.</p>
  
  <p>My third thought was to split the file into smaller pieces. To be precise into 26 pieces (there are 26 letters in Latin language), each containing only entries starting for the same letter, e.g. in a.xml all entries that titles starts at "A" letter. Final files would be like tens of MB, max around 200 MB I think. But there's the same problem with reading whole file. </p>
  
  <p>To read the file I used probably the fastest way: using <code>StreamReader</code>. I read somewhere that <code>StreamReader</code> and <code>XmlReader</code> class from <code>System.Xml</code> are the fastest methods. <code>StreamReader</code> even faster that <code>XmlReader</code>. It's obvious that I can't load all this file into memory. I have installed only 3GB of RAM and Win7 takes like 800MB-1GB when fully loaded.</p>
  
  <p>So I'm asking for help. What is the best to do. The point is that search this XML file has to be fast. Has to be faster then downloading single Wikipedia pages in HTML format. I'm not even sure if that is possible.</p>
  
  <p>Maybe load all the needed content into database? Maybe that would be faster? But still I will need to read the whole file as least once.</p>
  
  <p>I'm not sure if there are some limits about 1 question length, but I will put here also a sample of my indexing source code.</p>
  
  <pre><code>while (reading)
  {
      if (!reader.EndOfStream)
      {
          line = reader.ReadLine();
          fileIndex += enc.GetByteCount(line) + 2; //+2 - to cover characters \r\n not included into line
          position = 0;
      }
      else
      {
          reading = false;
          continue;
      }
  
      if (currentArea == Area.nothing)    //nothing interesting at the moment
      {
           //search for position of &lt;title&gt; tag
           position = MoveAfter("&amp;lt;title&gt;", line, position);    //searches until it finds &amp;lt;title&gt; tag
           if (position &gt;= 0) currentArea = Area.title;
           else continue;
      }
  
      (...)
  
      if (currentArea == Area.text)
      {
           position = MoveAfter("&amp;lt;text", line, position);
           if (position &gt;= 0)
           {
                long index = fileIndex;
                index -= line.Length;
                WriteIndex(currentTitle, currentId, index);
                currentArea = Area.nothing;
           }
           else continue;
       }
   }
  
   reader.Close();
   reader.Dispose();
   writer.Close();
   }
  
   private void WriteIndex(string title, string id, long index)
   {
       writer.WriteLine(title + ";" + id + ";" + index.ToString());
   }
  </code></pre>
  
  <p>Best Regards and Thanks in advance,</p>
  
  <p>ventus</p>
  
  <p>Edit: Link to this Wiki's dump <a href="http://download.wikimedia.org/plwiki/20100629/" rel="nofollow noreferrer">http://download.wikimedia.org/plwiki/20100629/</a></p>