Saving Large Spark ML Pipeline to HDFS I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum message size: 

    scala&gt; val mod = pipeline.fit(df)
    mod: org.apache.spark.ml.PipelineModel = pipeline_936bcade4716
    scala&gt; mod.write.overwrite().save(modelPath.concat("model"))
    18/01/08 10:00:32 WARN TaskSetManager: Stage 8 contains a task of very large size 
    (755610 KB). The maximum recommended task size is 100 KB.
    org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 
    2606:0 was 777523713 bytes, which exceeds max allowed: spark.rpc.message.maxSize 
    (134217728 bytes). Consider increasing spark.rpc.message.maxSize 
    or using broadcast variables for large values.


Making the following assumptions about the problem area:


It's not possible to decrease the size of the model  AND
It's not possible to increase the maximum message size to a point where the pipeline would fit in a single message.


Are there any methods that would allow me to save the pipeline successfully to HDFS?
You can do the following:
A Pipeline can be made of other pipelines. Isn't that great?
A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a Transformer.
This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform on the DataFrame.