title: Train NER model avoiding outofmemory error
body: <p>I am trying to train a NER model using the <a href="https://nlp.stanford.edu/software/CRF-NER.shtml#Download" rel="noreferrer">stanford-nlp library</a>. I have a file in the needed format with a size of 6GB. The configuration file:</p>
  
  <pre><code>usePrevSequences=true
  useClassFeature=true
  useTypeSeqs2=true
  useSequences=true
  wordShape=chris2useLC
  useTypeySequences=true
  useDisjunctive=true
  noMidNGrams=true
  serializeTo=ner-model50.ser.gz
  maxNGramLeng=6
  useNGrams=true
  usePrev=true
  useNext=true
  maxLeft=1
  trainFile=trainData
  map=word=0,answer=1
  useWord=true
  useTypeSeqs=true
  </code></pre>
  
  <p>Every sentence is splitted by empty line as suggested in the documentation. So when I run the command with <code>mx25g</code>: </p>
  
  <pre><code>java -mx25g -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop config.prop
  </code></pre>
  
  <p>I got the error: </p>
  
  <blockquote>
    <p>Exception in thread "main" java.lang.OutOfMemoryError: Java heap space</p>
  </blockquote>
  
  <p>From the <a href="https://nlp.stanford.edu/software/parser-faq.shtml#k" rel="noreferrer">FAQ page</a> of the library it is suggested to split the file to multiple files in case of memory error.</p>
  
  <p>To do that you need to change the config file by using the <code>trainFileList</code> option instead of <code>trainFile</code> option. Now I have multiple files of around 250MB each one. I got the following error running the same command:</p>
  
  <blockquote>
    <p>Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded</p>
  </blockquote>
  
  <p>So it seems that splitting the files didnt help. Do you think spliting to even smaller files will help? or it doesnt matter?Any way to overpass that problem? Any insight is useful. Thanks in advance.</p>