title: How do I create a 'word stream' and 'document stream' in Python?
body: <p>I want to take a bunch of text files and combine them all into two arrays - a 'word stream' and a 'document stream'. This is done by counting the total number of word tokens in the corpus then creating the arrays in which each entry in the word stream corresponds to the word associated with that token and the document stream corresponds to the document the word came from. </p>
  
  <p>For example, if the corpus is</p>
  
  <pre><code>Doc1: "The cat sat on the mat"
  Doc2: "The fox jumped over the dog"
  </code></pre>
  
  <p>The word stream will look like this:</p>
  
  <pre><code>WS: 1 2 3 4 1 5 1 6 7 8 1 9
  DS: 1 1 1 1 1 1 2 2 2 2 2 2 
  </code></pre>
  
  <p>I'm not exactly sure how to go about doing this, so my question is essentially this: How do I convert a text file into an array of word tokens? </p>