title: How to perform Name Entity Recognition faster on large files in using Python
body: <p>I am trying to perform Name Entity Recognition on large files such as text files as large as 1GB that is about 9 million lines of text.</p>
  <p>I have the below code that prints N.E.R outputs. So I basically do <code>python3 NER.py &gt; output.txt</code>. However this takes so much time and doesn't even finish when I ran it for 3 hours now. How can I make this usable?</p>
  <pre><code>import nltk
  import re
  
  sentence = &quot;&quot;
  
  with open(&quot;input.txt&quot;, &quot;r&quot;) as inFile:
      sentence = inFile.read()
  
  for sent in nltk.sent_tokenize(sentence):
      for chunk in nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(sent))):
          if hasattr(chunk, 'label'):
              print(chunk.label(), ' '.join(c[0] for c in chunk))
  </code></pre>