title: Set thresholds in PySpark multinomial logistic regression
body: <p>I would like to perform a multinomial logistic regression but I can't set <code>threshold</code> and <code>thresholds</code> parameters correctly. Consider the following DF:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.ml.linalg import DenseVector
  
  test_train_df = (
  sqlc
  .createDataFrame([(0, DenseVector([-1.0, 1.2, 0.7])),
                    (0, DenseVector([3.1, -2.0, -2.9])),
                    (1, DenseVector([1.0, 0.8, 0.3])),
                    (1, DenseVector([4.2, 1.4, -1.7])),
                    (0, DenseVector([-1.9, 2.5, -2.3])),
                    (2, DenseVector([2.6, -0.2, 0.2])),
                    (1, DenseVector([0.3, -3.4, 1.8])),
                    (2, DenseVector([-1.0, -3.5, 4.7]))],
                   ['label', 'features'])
  )
  </code></pre>
  
  <p>My label has 3 classes, so I have to set <code>thresholds</code> (plural, which default is <code>None</code>) rather than <code>threshold</code> (singular, which default is <code>0.5</code>). Then I write:</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.ml import classification as cl
  
  test_logit_abst = (
      cl.LogisticRegression()
      .setFamily('multinomial')
      .setThresholds([.5, .5, .5])
  )
  </code></pre>
  
  <p>Then I would like to fit the model on my DF:</p>
  
  <pre class="lang-python prettyprint-override"><code>test_logit = test_logit_abst.fit(test_train_df)
  </code></pre>
  
  <p>but when executing this last command I get an error:</p>
  
  <pre class="lang-python prettyprint-override"><code>---------------------------------------------------------------------------
  Py4JJavaError                             Traceback (most recent call last)
  ~/anaconda3/lib/python3.6/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
       62         try:
  ---&gt; 63             return f(*a, **kw)
       64         except py4j.protocol.Py4JJavaError as e:
  
  ~/anaconda3/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
      318                     "An error occurred while calling {0}{1}{2}.\n".
  --&gt; 319                     format(target_id, ".", name), value)
      320             else:
  
  Py4JJavaError: An error occurred while calling o3769.fit.
  : java.lang.IllegalArgumentException: requirement failed: Logistic Regression found inconsistent values for threshold and thresholds.  Param threshold is set (0.5), indicating binary classification, but Param thresholds is set with length 3. Clear one Param value to fix this problem.
  
  During handling of the above exception, another exception occurred:
  
  IllegalArgumentException                  Traceback (most recent call last)
  &lt;ipython-input-211-8f3443f41b6b&gt; in &lt;module&gt;()
  ----&gt; 1 test_logit = test_logit_abst.fit(test_train_df)
  
  ~/anaconda3/lib/python3.6/site-packages/pyspark/ml/base.py in fit(self, dataset, params)
       62                 return self.copy(params)._fit(dataset)
       63             else:
  ---&gt; 64                 return self._fit(dataset)
       65         else:
       66             raise ValueError("Params must be either a param map or a list/tuple of param maps, "
  
  ~/anaconda3/lib/python3.6/site-packages/pyspark/ml/wrapper.py in _fit(self, dataset)
  263
      264     def _fit(self, dataset):
  --&gt; 265         java_model = self._fit_java(dataset)
      266         return self._create_model(java_model)
  267
  
  ~/anaconda3/lib/python3.6/site-packages/pyspark/ml/wrapper.py in _fit_java(self, dataset)
      260         """
      261         self._transfer_params_to_java()
  --&gt; 262         return self._java_obj.fit(dataset._jdf)
  263
      264     def _fit(self, dataset):
  
  ~/anaconda3/lib/python3.6/site-packages/py4j/java_gateway.py in __call__(self, *args)
     1131         answer = self.gateway_client.send_command(command)
     1132         return_value = get_return_value(
  -&gt; 1133             answer, self.gateway_client, self.target_id, self.name)
  1134
     1135         for temp_arg in temp_args:
  
  ~/anaconda3/lib/python3.6/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
       77                 raise QueryExecutionException(s.split(': ', 1)[1], stackTrace)
       78             if s.startswith('java.lang.IllegalArgumentException: '):
  ---&gt; 79                 raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
       80             raise
       81     return deco
  
  IllegalArgumentException: 'requirement failed: Logistic Regression found inconsistent values for threshold and thresholds.  Param threshold is set (0.5), indicating binary classification, but Param thresholds is set with length 3. Clear one Param value to fix this problem.'
  </code></pre>
  
  <p>The error says <code>threshold</code> is set. This looks strange, as the <a href="http://spark.apache.org/docs/2.1.0/api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegression.setThresholds" rel="nofollow noreferrer">documentation</a> says that setting <code>thresholds</code> (plural) clears <code>threshold</code> (singular), so that the value <code>0.5</code> should be deleted.
  So, how to clear <code>threshold</code> since no <code>clearThreshold()</code> exists?</p>
  
  <p>In order to achieve this I tried to clear <code>threshold</code> this way:</p>
  
  <pre class="lang-python prettyprint-override"><code>logit_abst = (
      cl.LogisticRegression()
      .setFamily('multinomial')
      .setThresholds([.5, .5, .5])
      .setThreshold(None)
  )
  </code></pre>
  
  <p>This time the fit command works, I even obtain the model intercept and coefficients:</p>
  
  <pre class="lang-python prettyprint-override"><code>test_logit.interceptVector
  DenseVector([65.6445, 31.6369, -97.2814])
  
  test_logit.coefficientMatrix
  DenseMatrix(3, 3, [-76.4534, -19.4797, -79.4949, 12.3659, 4.642, 4.1057, 64.0876, 14.8377, 75.3892], 1)
  </code></pre>
  
  <p>But if I try to get <code>thresholds</code> (plural) from <code>test_logit_abst</code> I get an error:</p>
  
  <pre class="lang-python prettyprint-override"><code>test_logit_abst.getThresholds()
  
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-214-fc1c8617ce80&gt; in &lt;module&gt;()
  ----&gt; 1 test_logit_abst.getThresholds()
  
  ~/anaconda3/lib/python3.6/site-packages/pyspark/ml/classification.py in getThresholds(self)
      363         if not self.isSet(self.thresholds) and self.isSet(self.threshold):
      364             t = self.getOrDefault(self.threshold)
  --&gt; 365             return [1.0-t, t]
      366         else:
      367             return self.getOrDefault(self.thresholds)
  
  TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
  </code></pre>
  
  <p>What does this mean?</p>
  
  <hr>
  
  <p>As a further detail, curiously (and incomprehensibly to me) inverting the order of the parameters settings produces the first error I posted above:</p>
  
  <pre class="lang-python prettyprint-override"><code>logit_abst = (
      cl.LogisticRegression()
      .setFamily('multinomial')
      .setThreshold(None)
      .setThresholds([.5, .5, .5])
  )
  </code></pre>
  
  <p>Why does changing the order of the "set" instructions change the output as well?</p>