title: Python compute cosine similarity on two directories of files
body: <p>I have two directories of files. One contains human-transcribed files and the other contains IBM Watson transcribed files. Both directories have the same number of files, and both were transcribed from the same telephony recordings.</p>
  <p>I'm computing cosine similarity using SpaCy's .similarity between the matching files and print or store the result along with the compared file names. I have attempted using a function to iterate through in addition to for loops but cannot find a way to iterate between both directories, compare the two files with a matching index, and print the result.</p>
  <p>Here's my current code:</p>
  <pre><code># iterate through files in both directories
  for human_file, api_file in os.listdir(human_directory), os.listdir(api_directory):
      # set the documents to be compared and parse them through the small spacy nlp model
      human_model = nlp_small(open(human_file).read())
      api_model = nlp_small(open(api_file).read())
      
      # print similarity score with the names of the compared files
      print(&quot;Similarity using small model:&quot;, human_file, api_file, human_model.similarity(api_model))
  </code></pre>
  <p>I've gotten it to work with iterating through just one directory and checked that it has the expected output by printing the file name, but it doesn't work when using both directories. I've also tried something like this:</p>
  <pre><code># define directories
  human_directory = os.listdir(&quot;./00_data/Human Transcripts&quot;)
  api_directory = os.listdir(&quot;./00_data/Watson Scripts&quot;)
  
  # function for cosine similarity of files in two directories using small model
  def nlp_small(human_directory, api_directory):
      for i in (0, (len(human_directory) - 1)):
          print(human_directory[i], api_directory[i])
  
  nlp_small(human_directory, api_directory)
  </code></pre>
  <p>Which returns:</p>
  <pre><code>human_10.txt watson_10.csv
  human_9.txt watson_9.csv
  </code></pre>
  <p>But that's only two of the files, not all 17 of them.</p>
  <p>Any pointers on iterating through a matching index on both directories would be much appreciated.</p>
  <p>EDIT:
  Thanks to @kevinjiang, here's the working code block:</p>
  <pre><code># set the directories containing transcripts
  human_directory = os.path.join(os.getcwd(), &quot;00_data\Human Transcripts&quot;)
  api_directory = os.path.join(os.getcwd(), &quot;00_data\Watson Scripts&quot;)
  
  # iterate through files in both directories
  for human_file, api_file in zip(os.listdir(human_directory), os.listdir(api_directory)):
      # set the documents to be compared and parse them through the small spacy nlp model
      human_model = nlp_small(open(os.path.join(os.getcwd(), &quot;00_data\Human Transcripts&quot;, human_file)).read())
      api_model = nlp_small(open(os.path.join(os.getcwd(), &quot;00_data\Watson Scripts&quot;, api_file)).read())
      
      # print similarity score with the names of the compared files
      print(&quot;Similarity using small model:&quot;, human_file, api_file, human_model.similarity(api_model))
  </code></pre>
  <p>And here's most of the output (need to fix a UTF-16 character in one of the files that halts the loop):</p>
  <pre><code>nlp_small = spacy.load('en_core_web_sm')
  Similarity using small model: human_10.txt watson_10.csv 0.9274665883462793
  Similarity using small model: human_11.txt watson_11.csv 0.9348740684005554
  Similarity using small model: human_12.txt watson_12.csv 0.9362025469343344
  Similarity using small model: human_13.txt watson_13.csv 0.9557355330988958
  Similarity using small model: human_14.txt watson_14.csv 0.9088701120190216
  Similarity using small model: human_15.txt watson_15.csv 0.9479464053189846
  Similarity using small model: human_16.txt watson_16.csv 0.9599724037676819
  Similarity using small model: human_17.txt watson_17.csv 0.9367605599306302
  Similarity using small model: human_18.txt watson_18.csv 0.8760760037870665
  Similarity using small model: human_2.txt watson_2.csv 0.9184563762823503
  Similarity using small model: human_3.txt watson_3.csv 0.9287452822270265
  Similarity using small model: human_4.txt watson_4.csv 0.9415664367046419
  Similarity using small model: human_5.txt watson_5.csv 0.9158895909429551
  Similarity using small model: human_6.txt watson_6.csv 0.935313240861153
  </code></pre>
  <p>After I've fixed the character encoding bug I'll be wrapping it in a function so that I can call the large or small model on two directories for the remaining APIs I have to test.</p>