title: LDA in Spark 1.3.1. Converting raw data into Term Document Matrix?
body: <p>I'm trying out LDA with Spark 1.3.1 in Java and got this error:</p>
  
  <pre><code>Error: application failed with exception
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NumberFormatException: For input string: "��"
  </code></pre>
  
  <p>My .txt file looks like this: 
   put weight find difficult pull ups push ups now 
   blindness diseases everything eyes work perfectly except ability take light use light form images 
   role model kid 
    Dear recall saddest memory childhood</p>
  
  <p>This is the code:</p>
  
  <pre><code>import scala.Tuple2;
  
  import org.apache.spark.api.java.*;
  import org.apache.spark.api.java.function.Function;
  import org.apache.spark.mllib.clustering.LDAModel;
  import org.apache.spark.mllib.clustering.LDA;
  import org.apache.spark.mllib.linalg.Matrix;
  import org.apache.spark.mllib.linalg.Vector;
  import org.apache.spark.mllib.linalg.Vectors;
  import org.apache.spark.SparkConf;
  
  public class JavaLDA {
    public static void main(String[] args) {
      SparkConf conf = new SparkConf().setAppName("LDA Example");
      JavaSparkContext sc = new JavaSparkContext(conf);
  
      // Load and parse the data
      String path = "/tutorial/input/askreddit20150801.txt";
      JavaRDD&lt;String&gt; data = sc.textFile(path);
      JavaRDD&lt;Vector&gt; parsedData = data.map(
          new Function&lt;String, Vector&gt;() {
            public Vector call(String s) {
              String[] sarray = s.trim().split(" ");
              double[] values = new double[sarray.length];
              for (int i = 0; i &lt; sarray.length; i++)
                values[i] = Double.parseDouble(sarray[i]);
              return Vectors.dense(values);
            }
          }
      );
      // Index documents with unique IDs
      JavaPairRDD&lt;Long, Vector&gt; corpus = JavaPairRDD.fromJavaRDD(parsedData.zipWithIndex().map(
          new Function&lt;Tuple2&lt;Vector, Long&gt;, Tuple2&lt;Long, Vector&gt;&gt;() {
            public Tuple2&lt;Long, Vector&gt; call(Tuple2&lt;Vector, Long&gt; doc_id) {
              return doc_id.swap();
            }
          }
      ));
      corpus.cache();
  
      // Cluster the documents into three topics using LDA
      LDAModel ldaModel = new LDA().setK(100).run(corpus);
  
      // Output topics. Each is a distribution over words (matching word count vectors)
      System.out.println("Learned topics (as distributions over vocab of " + ldaModel.vocabSize()
          + " words):");
      Matrix topics = ldaModel.topicsMatrix();
      for (int topic = 0; topic &lt; 100; topic++) {
        System.out.print("Topic " + topic + ":");
        for (int word = 0; word &lt; ldaModel.vocabSize(); word++) {
          System.out.print(" " + topics.apply(word, topic));
        }
        System.out.println();
      }
  
      ldaModel.save(sc.sc(), "myLDAModel");
  
    }
  }
  </code></pre>
  
  <p>Anyone know why this happened? I'm just trying LDA Spark for the first time. Thanks.</p>