title: Memory efficient algorithm to build word occurrence index of a huge text file in C++
body: <p>I have to write a program that reads a huge file with with operator>>, count things and then insert the whole in multimap.</p>
  
  <p>It works perfectly with small files but with big files the program runs out of memory. It throws a bad_alloc on Windows 32-bit.</p>
  
  <p>I made a 64-bit build on Windows (in order to avoid the memory allocation limit for 32-bit programs on Windows), opened the task manager and what I see is that the memory is increasing continually, and when the RAM is full the program crashes.
  Do you know how to deal with this ?</p>
  
  <p>Here is the code : </p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;string&gt;
  #include &lt;fstream&gt;
  #include &lt;unordered_map&gt;
  
  using namespace std;
  
  int main()
  {
      ifstream fin("C:/text.txt");
      string const outfile("C:/out/out.txt");
      ofstream fout (outfile.c_str());
      string word;
      unordered_multimap&lt;string, int&gt; mymm;
      int intnum = 0;
  
      while(fin &gt;&gt; word)
      {
          mymm.insert(pair&lt;string, int&gt;(word, intnum));
          ++intnum;
      }
  
      return 0;
  }
  </code></pre>