title: NLTK tokenize - faster way?
body: <p>I have a method that takes in a String parameter, and uses NLTK to break the String down to sentences, then into words. Afterwards, it converts each word into lowercase, and finally creates a dictionary of the frequency of each word.</p>
  
  <pre><code>import nltk
  from collections import Counter
  
  def freq(string):
      f = Counter()
      sentence_list = nltk.tokenize.sent_tokenize(string)
      for sentence in sentence_list:
          words = nltk.word_tokenize(sentence)
          words = [word.lower() for word in words]
          for word in words:
              f[word] += 1
      return f
  </code></pre>
  
  <p>I'm supposed to optimize the above code further to result in faster preprocessing time, and am unsure how to do so. The return value should obviously be exactly the same as the above, so I'm expected to use nltk though not explicitly required to do so.</p>
  
  <p>Any way to speed up the above code? Thanks.</p>