title: Create many Spark MLlib models based on partitioned DataFrame using Pipeline
body: <p><code>scala&gt; spark.version
  res8: String = 2.2.0</code></p>
  
  <p>I'm working with a spark Dataframe that contains a column <code>locationID</code>. I've created a MLlib pipeline to build a linear regression model and it works when I feed it data for a single <code>locationID</code>. I would now like to create many models for each 'locationID' (there may be a few thousand locationID's in production). I would like to save the model coefficients for each model.</p>
  
  <p>I'm not sure how this can be done in Scala.</p>
  
  <p>My pipeline is defined like this:</p>
  
  <pre><code>import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer}
  import org.apache.spark.ml.regression.LinearRegression
  import org.apache.spark.ml.regression.LinearRegressionModel
  import org.apache.spark.ml.feature.VectorAssembler
  import org.apache.spark.ml.{Pipeline, PipelineModel}
  import org.apache.spark.sql
  
  
  // Load the regression input data
  val mydata = spark.read.format("csv")
    .option("header", "true")
    .option("inferSchema", "true")
    .load("./inputdata.csv")
  
  // Crate month one hot encoding
  val monthIndexer = new StringIndexer()
    .setInputCol("month")
    .setOutputCol("monthIndex").fit(mydata)
  val monthEncoder = new OneHotEncoder()
    .setInputCol(monthIndexer.getOutputCol)
    .setOutputCol("monthVec")
  val assembler =  new VectorAssembler()
    .setInputCols(Array("monthVec","tran_adr"))
    .setOutputCol("features")
  val lr = new LinearRegression()
    .setMaxIter(10)
    .setRegParam(0.3)
    .setElasticNetParam(0.8)
  val pipeline = new Pipeline()
    .setStages(Array(monthIndexer, monthEncoder, assembler, lr))
  
  
  // Fit using the model pipeline
  val myPipelineModel = pipeline.fit(mydata)
  </code></pre>
  
  <p>I then am able to pull the model details like this:</p>
  
  <pre><code>val modelExtract = myPipelineModel.stages(3).asInstanceOf[LinearRegressionModel]
  
  println(s"Coefficients: ${modelExtract.coefficients} Intercept: ${modelExtract.intercept}")
  // Summarize the model over the training set and print out some metrics
  val trainingSummary = modelExtract.summary
  println(s"numIterations: ${trainingSummary.totalIterations}")
  println(s"objectiveHistory: [${trainingSummary.objectiveHistory.mkString(",")}]")
  trainingSummary.residuals.show()
  println(s"RMSE: ${trainingSummary.rootMeanSquaredError}")
  println(s"r2: ${trainingSummary.r2}")
  </code></pre>
  
  <p>Now I want to group on a column <code>locationID</code> found in <code>mydata</code> and run the pipeline on each partition of the data.</p>
  
  <p>I've tried using using groupby but I can only aggregate.</p>
  
  <pre><code>val grouped = mydata.groupBy("locationID")
  </code></pre>
  
  <p>I've also tried to pull the unique <code>locationID</code> as a list and loop through it:</p>
  
  <pre><code>val locationList = mydata.select(mydata("prop_code")).distinct
  
  locationList.foreach { printLn }
  </code></pre>
  
  <p>I know that spark is not ideal for creating many smaller models, and it best for creating one model on a large set of data, but I've been tasked to do this as a proof of concept. </p>
  
  <p>What is the correct approach for doing something like this in spark?</p>