title: Can't figure out cause of Spark LinearRegression Error
body: <p>I'm trying to do a very simple <code>LinearRegression</code> in PySpark using a housing data set I found on Kaggle. There are a bunch of columns, but to make this (virtually) as simple as possible, I'm retaining just two of the columns (after starting with all of them), and still no luck getting a model trained. Here is what the data frame looks like before going through the regression step:</p>
  
  <pre><code>2016-09-07 17:12:08,804 root INFO [Row(price=78000.0, sqft_living=780.0, sqft_lot=16344.0, features=DenseVector([780.0, 16344.0])), Row(price=80000.0, sqft_living=430.0, sqft_lot=5050.0, features=DenseVector([430.0, 5050.0])), Row(price=81000.0, sqft_living=730.0, sqft_lot=9975.0, features=DenseVector([730.0, 9975.0])), Row(price=82000.0, sqft_living=860.0, sqft_lot=10426.0, features=DenseVector([860.0, 10426.0])), Row(price=84000.0, sqft_living=700.0, sqft_lot=20130.0, features=DenseVector([700.0, 20130.0])), Row(price=85000.0, sqft_living=830.0, sqft_lot=9000.0, features=DenseVector([830.0, 9000.0])), Row(price=85000.0, sqft_living=910.0, sqft_lot=9753.0, features=DenseVector([910.0, 9753.0])), Row(price=86500.0, sqft_living=840.0, sqft_lot=9480.0, features=DenseVector([840.0, 9480.0])), Row(price=89000.0, sqft_living=900.0, sqft_lot=4750.0, features=DenseVector([900.0, 4750.0])), Row(price=89950.0, sqft_living=570.0, sqft_lot=4080.0, features=DenseVector([570.0, 4080.0]))]
  </code></pre>
  
  <p>I'm using the following code to train the model:</p>
  
  <pre><code>    standard_scaler = StandardScaler(inputCol='features',
                                       outputCol='scaled')
      lr = LinearRegression(featuresCol=standard_scaler.getOutputCol(), labelCol='price', weightCol=None,
                            maxIter=100, tol=1e-4)
      pipeline = Pipeline(stages=[standard_scaler, lr])
      grid = (ParamGridBuilder()
              .baseOn({lr.labelCol: 'price'})
              .addGrid(lr.regParam, [0.1, 1.0])
              .addGrid(lr.elasticNetParam, elastic_net_params or [0.0, 1.0])
              .build())
      ev = RegressionEvaluator(metricName="rmse", labelCol='price')
      cv = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=grid,
                          evaluator=ev,
                          numFolds=5)
      model = cv.fit(data).bestModel
  </code></pre>
  
  <p>The error I'm getting is:</p>
  
  <pre><code>2016-09-07 17:12:08,805 root INFO Training regression model...
  2016-09-07 17:12:09,530 root ERROR An error occurred while calling o60.fit.
  : java.lang.NullPointerException
      at org.apache.spark.ml.regression.LinearRegression.train(LinearRegression.scala:164)
      at org.apache.spark.ml.regression.LinearRegression.train(LinearRegression.scala:70)
      at org.apache.spark.ml.Predictor.fit(Predictor.scala:90)
      at org.apache.spark.ml.Predictor.fit(Predictor.scala:71)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:211)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>Any thoughts?</p>