title: Load Pyspark.ml model from S3 using Pipeline
body: <p>I am trying to save a trained model to S3 storage and then trying to load and predict using this model via Pipeline package from  pyspark.ml.
  Here's an example of how I am saving my model.</p>
  <pre><code>#stage_1 to stage_4 are some basic trasnformation on data one-hot encoding e.t.c
  # define stage 5: logistic regression model                          
   stage_5 = LogisticRegression(featuresCol='features',labelCol='label')
  
   # SETUP THE PIPELINE
   regression_pipeline = Pipeline(stages= [stage_1, stage_2, stage_3, stage_4, stage_5])
  
   # fit the pipeline for the trainind data
   model = regression_pipeline.fit(dataFrame1)
  
   model_path =&quot;s3://s3-dummy_path-orch/dummy models/pipeline_testing_1.model&quot;
   model.save(model_path)
  
  </code></pre>
  <p>I am able to save the model successfully &amp; at the above mentioned model path two folders get created</p>
  <ol>
  <li>stages</li>
  <li>metadata.</li>
  </ol>
  <p>However when I am trying to load the model it is giving me the below error.</p>
  <pre><code>Traceback (most recent call last):
    File &quot;/tmp/pythonScript_85ff2462_e087_4805_9f50_0c75fc4302e2958379757178872310.py&quot;, line 75, in &lt;module&gt;
      pipelineModel = Pipeline.load(model_path)
    File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/util.py&quot;, line 362, in load
    File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py&quot;, line 207, in load
    File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/util.py&quot;, line 300, in load
    File &quot;/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py&quot;, line 1257, in __call__
    File &quot;/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py&quot;, line 79, in deco
  pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.Pipeline but found class name org.apache.spark.ml.PipelineModel'
  </code></pre>
  <p>I am trying to load the model as below:</p>
  <pre><code>from pyspark.ml import Pipeline
  
  ## same path used while #model.save in the above code snippet
  model_path =&quot;s3://s3-dummy_path-orch/dummy models/pipeline_testing_1.model&quot; 
  
  pipelineModel = Pipeline.load(model_path)
  
  </code></pre>
  <p>How could I go about rectifying this?</p>