title: Using directory as input for tf-idf with python `textblob`
body: <p>I am trying to adapt this code (source found <a href="http://stevenloria.com/finding-important-words-in-a-document-using-tf-idf/" rel="nofollow noreferrer">here</a>)to iterate through a directory of files, instead of having the input hard-coded. </p>
  
  <pre><code>#!/usr/bin/python
  # -*- coding: utf-8 -*-
  
  from __future__ import division, unicode_literals
  import math
  from textblob import TextBlob as tb
  
  def tf(word, blob):
      return blob.words.count(word) / len(blob.words)
  
  def n_containing(word, bloblist):
      return sum(1 for blob in bloblist if word in blob)
  
  def idf(word, bloblist):
      return math.log(len(bloblist) / (1 + n_containing(word, bloblist)))
  
  def tfidf(word, blob, bloblist):
      return tf(word, blob) * idf(word, bloblist)
  
  
  document1 = tb("""Today, the weather is 30 degrees in Celcius. It is really hot""")
  
  document2 = tb("""I can't believe the traffic headed to the beach. It is really a circus out there.'""")
  
  document3 = tb("""There are so many tolls on this road. I recommend taking the interstate.""")
  
  bloblist = [document1, document2, document3]
  for i, blob in enumerate(bloblist):
      print("Document {}".format(i + 1))
      scores = {word: tfidf(word, blob, bloblist) for word in blob.words}
      sorted_words = sorted(scores.items(), key=lambda x: x[1], reverse=True)
      for word, score in sorted_words:
          score_weight = score * 100 
          print("\t{}, {}".format(word, round(score_weight, 5)))
  </code></pre>
  
  <p>I would like to use an an input txt files in a directory, rather than each hard-coded <code>document</code>.</p>
  
  <p>For instance, imagine I had a directory <code>foo</code> which contains three files <code>file1</code>, <code>file2</code>, <code>file3</code>.</p>
  
  <p>File 1 contains the contents that <code>document1</code> contains, i.e.</p>
  
  <p>file1:</p>
  
  <pre><code>Today, the weather is 30 degrees in Celcius. It is really hot
  </code></pre>
  
  <p>File 2 contains the contents that <code>document2</code> contains, i.e.</p>
  
  <pre><code>I can't believe the traffic headed to the beach. It is really a circus out there.
  </code></pre>
  
  <p>File 3 contains the contents that <code>document3</code> contains, i.e.</p>
  
  <pre><code>There are so many tolls on this road. I recommend taking the interstate.
  </code></pre>
  
  <p>I have though to use <code>glob</code> to achieve my desired result, and I have come up with the following code adapation, which correctly identifies the files, but does not process them individually, as the original code does:</p>
  
  <pre><code>file_names = glob.glob("/path/to/foo/*")
  files =  map(open,file_names)
  documents = [file.read() for file in files]
  [file.close() for file in files]
  
  
  bloblist = [documents]
  for i, blob in enumerate(bloblist):
      print("Document {}".format(i + 1))
      scores = {word: tfidf(word, blob, bloblist) for word in blob.words}
      sorted_words = sorted(scores.items(), key=lambda x: x[1], reverse=True)
      for word, score in sorted_words:
          score_weight = score * 100 
          print("\t{}, {}".format(word, round(score_weight, 5)))
  </code></pre>
  
  <p>How can I maintain the scores for each individual file using <code>glob</code>?</p>
  
  <p>The desired result after using the files in a directory as input would be the same as the original code [results truncuated to top 3 for space]:</p>
  
  <pre><code>Document 1
      Celcius, 3.37888
      30, 3.37888
      hot, 3.37888
  Document 2
      there, 2.38509
      out, 2.38509
      headed, 2.38509
  Document 3
      on, 3.11896
      this, 3.11896
      many, 3.11896
  </code></pre>
  
  <p>A similar question <a href="https://stackoverflow.com/questions/22434092/compute-tf-idf-with-corpus">here</a> did not fully solve the problem. I was wondering how I can call the files to calculate the <code>idf</code> but maintain them separately for calculate the full <code>tf-idf</code>?</p>