title: how to manage a JavaRDD<LabeledPoint> data structure in Spark
body: <p>I have a data file like this:</p>
  
  <pre><code>"id"; "a"; "b"; "c"; "d"; "e"; "f"; "outcome"
  1144216938;1988;0;0;1;1965.115431348724;19.07168894289186;0
  1093965329;2004;1;1;1;302.2244897959184;16.762951334379906;1
  ....
  </code></pre>
  
  <p>The first column represents the id label for each case, while the last column represents a binary outcome, either 0 or 1. All the variables in-between are used in a logistic regression model (I haven't normalised them yet). I read the file in Spark with:</p>
  
  <pre><code>JavaRDD&lt;String&gt; data = sc.textFile(filename);
  JavaRDD&lt;LabeledPoint&gt; parseddata = data
      .map((String line) -&gt; {
          String[] parts = line.split(";");
          double[] points = new double[parts.length - 1];
          for (int i = 0; i &lt; (parts.length - 1); i++) {
              points[i] = Double.valueOf(parts[i]);
          }
          return new LabeledPoint(Double.valueOf(parts[parts.length - 1]), Vectors.dense(points));
      });
  </code></pre>
  
  <p>I am unfamiliar with the LabeledPoint data structure, so this might be easy to solve, but I don't understand how... So I have three related questions:</p>
  
  <ul>
  <li>how to get rid of the file header? Right now I just eliminate it from the file: </li>
  </ul>
  
  <p><strong>EDIT</strong>
  as per eliasah comment, this should do the trick:</p>
  
  <pre><code>    // retrieve the header
      String first = data.first();
      // filter it out
      JavaRDD&lt;String&gt; filteredData = data.filter((String s) -&gt; {
         return !s.contains(first);
      });
  </code></pre>
  
  <p><strong>END EDIT</strong></p>
  
  <ul>
  <li><p>is there a way to map the first column as row names, so to exclude it from the points used in the regression model - apart from using:</p>
  
  <pre><code>for (int i = 1; i &lt; (parts.length - 1); i++) {
      points[i] = Double.valueOf(parts[i]);
  }
  </code></pre></li>
  <li><p>finally, how to map predictions to the right "id" when I retrieve the predictions from my regression model with:</p>
  
  <pre><code>JavaRDD&lt;Tuple2&lt;Object, Object&gt;&gt; predictionAndLabels = test.map((LabeledPoint p) -&gt; {
     Double prediction = model.predict(p.features());
     return new Tuple2&lt;Object, Object&gt;(prediction, p.label());
  });
  </code></pre></li>
  </ul>
  
  <p><strong>EDIT 2</strong>
  Despite the answer says it all, all I can come up with is:</p>
  
  <pre><code>JavaRDD&lt;Tuple2&lt;Object, Object&gt;&gt; results = filteredData.map((String line) -&gt; {
          String[] parts = line.split(";");
          double[] points = new double[parts.length - 1];
          //skip first column with npi values
          for (int i = 1; i &lt; (parts.length - 1); i++) {
              points[i] = Double.valueOf(parts[i]);
          }
          LabeledPoint labeledPoint = new LabeledPoint(Double.valueOf(parts[parts.length - 1]), Vectors.dense(points));
          return new Tuple2&lt;Object, Object&gt;(parts[0], model.predict(labeledPoint.features()));
      });
  </code></pre>
  
  <p>which is a pity because I am basically redoing the map again... is this the correct way?</p>
  
  <p><strong>EDIT 3</strong></p>
  
  <p>Thanks to other comments from eliasah, I now filter data like:</p>
  
  <pre><code>JavaRDD&lt;Tuple2&lt;String, LabeledPoint&gt;&gt; parsedData = filteredData.map((String line) -&gt; {
   String[] parts = line.split(";");
   double[] points = new double[parts.length - 1];
   for (int i = 1; i &lt; (parts.length - 1); i++) {
       points[i] = Double.valueOf(parts[i]);
   }
   return new Tuple2&lt;String, LabeledPoint&gt;(parts[0], new LabeledPoint(Double.valueOf(parts[parts.length - 1]), Vectors.dense(points)));
   });
  </code></pre>
  
  <p>I use this for training:</p>
  
  <pre><code>JavaRDD&lt;LabeledPoint&gt; training = parsedData
       .map((Tuple2&lt;String, LabeledPoint&gt; z) -&gt; {
           return z._2();
       });
  </code></pre>
  
  <p>And retrieve probabilities for each of my case with:</p>
  
  <pre><code>JavaRDD&lt;Tuple2&lt;String, Double&gt;&gt; results = parsedData
       .map((Tuple2&lt;String, LabeledPoint&gt; z) -&gt; {
           return new Tuple2&lt;String, Double&gt;(z._1(), model.predict(z._2().features()));
       });
  </code></pre>