title: Saving huge bigram dictionary to file using pickle
body: <p>a friend of mine wrote this little progam.
  the <code>textFile</code> is 1.2GB in size (7 years worth of newspapers).
  He successfully manages to create the dictionary but he cannot write it to a file using pickle(program hangs).</p>
  
  <pre><code>import sys
  import string
  import cPickle as pickle
  
  biGramDict = {}
  
  textFile = open(str(sys.argv[1]), 'r')
  biGramDictFile = open(str(sys.argv[2]), 'w')
  
  
  for line in textFile:
     if (line.find('&lt;s&gt;')!=-1):
        old = None
        for line2 in textFile:
           if (line2.find('&lt;/s&gt;')!=-1):
              break
           else:
              line2=line2.strip()
              if line2 not in string.punctuation:
                 if old != None:
                    if old not in biGramDict:
                       biGramDict[old] = {}
                    if line2 not in biGramDict[old]:
                       biGramDict[old][line2] = 0
                    biGramDict[old][line2]+=1
                 old=line2
  
  textFile.close()
  
  print "going to pickle..."    
  pickle.dump(biGramDict, biGramDictFile,2)
  
  print "pickle done. now load it..."
  
  biGramDictFile.close()
  biGramDictFile = open(str(sys.argv[2]), 'r')
  
  newBiGramDict = pickle.load(biGramDictFile)
  </code></pre>
  
  <p>thanks in advance.</p>
  
  <p><strong>EDIT</strong><br>
  for anyone interested i will briefly explain what this program does.
  assuming you have a file formated roughly like this:</p>
  
  <pre><code>&lt;s&gt;
  Hello
  ,
  World
  !
  &lt;/s&gt;
  &lt;s&gt;
  Hello
  ,
  munde
  !
  &lt;/s&gt;
  &lt;s&gt;
  World
  domination
  .
  &lt;/s&gt;
  &lt;s&gt;
  Total
  World
  domination
  !
  &lt;/s&gt;
  </code></pre>
  
  <ul>
  <li><code>&lt;s&gt;</code> are sentences separators. </li>
  <li>one word per line.</li>
  </ul>
  
  <p>a biGramDictionary is generated for later use.<br>
  something like this:</p>
  
  <pre><code>{
   "Hello": {"World": 1, "munde": 1}, 
   "World": {"domination": 2},
   "Total": {"World": 1},
  }
  </code></pre>
  
  <p>hope this helps. right now the strategy changed to using mysql because sqlite just wasn't working (probably because of the size)</p>