Model ensemble with Spark or Scikit Learn
As you said in your question, there is no way to do that with the baseline algorithms provided in MLLib.

Two ways you could do that is by either :


Creating a function to generate a Pipeline
Creating a Meta Estimator that would take your base learners and the forking column.


The first one is what you have stated in your question and the second one has been explained by Brian Spiering.

As you said making a custom Estimator and Transformer will make it works nicely with all the MLLib Tools as Tuning Tools.

If you want a really precise example how to do so, there is my library which implements meta algorithms for Ensemble Learning with Spark.

There is a way to train multiple estimators in parallel quite simply in the Stacking Classifier, and for all of them you have an example on how to ask for an estimator as the parameter (to chose either your logistic regression or decision tree).

You can pick ideas out of that !

Edit : Similar question on StackOverflow