title: When I run the spark sample for logistic regression, I get a perfect model. Did I screw up
body: <p>I'm running the example code from the <a href="https://spark.apache.org/docs/2.1.0/ml-classification-regression.html#binomial-logistic-regression" rel="nofollow noreferrer">spark docs</a> for logistic regression using pyspark and the attendant training summary code:</p>
  
  <pre><code>from pyspark.ml.classification import LogisticRegression
  
  # Load training data
  training = spark.read.format("libsvm").load("/user/tim/sample_svm/sample_libsvm_data.txt")
  
  lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
  
  # Fit the model
  lrModel = lr.fit(training)
  
  # Print the coefficients and intercept for logistic regression
  print("Coefficients: " + str(lrModel.coefficients))
  print("Intercept: " + str(lrModel.intercept))
  
  # We can also use the multinomial family for binary classification
  mlr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8, family="multinomial")
  
  # Fit the model
  mlrModel = mlr.fit(training)
  
  # Print the coefficients and intercepts for logistic regression with multinomial family
  print("Multinomial coefficients: " + str(mlrModel.coefficientMatrix))
  print("Multinomial intercepts: " + str(mlrModel.interceptVector))
  
  # Extract the summary from the returned LogisticRegressionModel instance trained
  # in the earlier example
  trainingSummary = lrModel.summary
  
  # Obtain the objective per iteration
  objectiveHistory = trainingSummary.objectiveHistory
  print("objectiveHistory:")
  for objective in objectiveHistory:
      print(objective)
  
  # Obtain the receiver-operating characteristic as a dataframe and areaUnderROC.
  trainingSummary.roc.show(500)
  print("areaUnderROC: " + str(trainingSummary.areaUnderROC))
  
  # Set the model threshold to maximize F-Measure
  fMeasure = trainingSummary.fMeasureByThreshold
  maxFMeasure = fMeasure.groupBy().max('F-Measure').select('max(F-Measure)').head()
  bestThreshold = fMeasure.where(fMeasure['F-Measure'] == maxFMeasure['max(F-Measure)']) \
      .select('threshold').head()['threshold']
  lr.setThreshold(bestThreshold)
  </code></pre>
  
  <p>and get:</p>
  
  <pre><code>areaUnderROC: 1.0
  </code></pre>
  
  <p>which I wouldn't expect. Perhaps it overfit and simply memorized the data, but I've done train and test, even randomized labels, and tweaked all the hyper parameters and they all led to the same thing:AUC=1.0. I tried the sample code for the SVC models, which uses the same dataset, and I get the same thing. </p>
  
  <p>I'd normally post the code, but I literally ran the example code only changing the path to the data file.  I've searched and searched and can find no example of anyone having run this example and examined the results. What's odd is that this dataset, sample_libsvm_data.txt, is used throughout the docs yet I can find neither analysis of it nor even an explanation of what the data actually is. </p>
  
  <p>As a result I've switched to using the RDD-based API of MLLIB because I can't make sense of the results of the sample code. I hope someone can tell me how I'm doing something wrong. </p>
  
  <p>EDIT: 
  As requested, here's the <a href="https://gist.github.com/timtiptoes/88bfda6be134b2482a8fa54264698f5a" rel="nofollow noreferrer">entire datafile</a>.</p>