How to deal with evaluations in a multi step ML pipeline
A pyspark.ml.PipelineModel is the result of calling the .fit() method of a pyspark.ml.Pipeline. This estimator is a sequence of Transformers and/or Estimators, all of which have a .transform() method. When you call the .transform() method of the pyspark.ml.PipelineModel-- say, when you want to make predictions using the trained model-- the .transform() method of each of the objects in the PipelineModel are called in sequence. 

In other words, the transformations for each step are applied by calling the .transform() method of the fitted pipeline- the API does this for you, or more accurately, gives you concise tools to do this as you need to.

What you're talking about is slightly outside the scope of this type of object, however. Train/test splitting occurs outside the Pipeline, and importantly so, for methodological reasons.

A PipelineModel couples data preparation steps with an estimator, but not training operations. Splitting the data, for example, is something you'd have to do outside the Pipeline.

The workflow might go:

Split the data into training and test datasets.

Define your data preparation steps and regression estimator, and combine these into a Pipeline.

Train the entire pipeline on your training data:

fit_pipeline = Pipeline.fit(train)


Make predictions on the training set:

train_preds = fit_pipeline.transform(train)


Use something like a pyspark.ml.evaluation.RegressionEvaluator to evaluate the performance on the training dataset:

evaluator = RegressionEvaluator()
evaluator.evaluate(train_preds)


Then, you can make predictions on the test dataset:

test_preds = fit_pipeline.transform(test)


And evaluate them:

evaluator.evaluate(train_preds)


But you have this completed model and data preparation object that can be used to make predictions on similarly structured data now, called fit_pipeline, which is an instance of pyspark.ml.PipelineModel.