title: How can I read a Lucene document field tokens after they are analyzed?
body: <p>If I create a document and add a field that is both stored and analyzed, how can I then read this field back as a list of tokens? I have the following:</p>
  
  <pre><code>            Document doc = new Document();
              doc.add(new Field("url", fileName, Store.YES, Index.NOT_ANALYZED));
              doc.add(new Field("text", fileContent, Store.YES, Index.ANALYZED));
              // add the document to the index
              writer.addDocument(doc);
  </code></pre>
  
  <p>So the fileContext is a String containing a lot of text. It is analyzed whereby it is tokenized when it is stored in the index. However, how can I get these tokens? I can retrieve the document from the index after it is stored, and I can read the "text" field from the document, but this is returned as a string. I would like to get the tokens if possible. My 'writer' is an IndexWriter instance and it uses a StandardAnalyzer. Any pointers would be very much welcomed. </p>
  
  <p>Thank you very much</p>