SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?
I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of predictions = model.transform(testData) as print(predictions) in PySpark you will get the probability of each labels, You can check the below code and output of the code:

from pyspark.sql import DataFrame
from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a Random Forest model.
rf = RandomForestClassifier(labelCol="label", featuresCol="features", numTrees=12,  maxDepth=10)

# Chain RF in a Pipeline
pipeline = Pipeline(stages=[rf])

# Train model.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)


Now your work start from here. try printing predictions and values of predictions

print(predictions)


Output:

DataFrame[label: double, features: vector, indexed: double, rawPrediction: vector, probability: vector, prediction: double]


So, in the DataFrame you have the probability which is the probability of each indexedLabel, Further I have checked it as:

print predictions.show(3)


Output:

+-----+--------------------+-------+--------------------+--------------------+----------+
|label|            features|indexed|       rawPrediction|         probability|prediction|
+-----+--------------------+-------+--------------------+--------------------+----------+
|  5.0|(2000,[141,260,50...|    0.0|[34.8672584923246...|[0.69734516984649...|       0.0|
|  5.0|(2000,[109,126,18...|    0.0|[34.6231572522266...|[0.69246314504453...|       0.0|
|  5.0|(2000,[185,306,34...|    0.0|[34.5016453103805...|[0.69003290620761...|       0.0|
+-----+--------------------+-------+--------------------+--------------------+----------+
only showing top 3 rows


Only for probability column:

print predictions.select('probability').take(2)


Output:

[Row(probability=DenseVector([0.6973, 0.1889, 0.0532, 0.0448, 0.0157])), Row(probability=DenseVector([0.6925, 0.1825, 0.0579, 0.0497, 0.0174]))]


In my case I have 5 indexedLabels and so the probability vector length is 5, Hope this will help you to get the probability of each labels in your problem.

P.S: You will probably get the probability in Decision Tree, Logistic regression. Just try to get the summary of model.transform(testData).