title: Using dplyr and broom to compute kmeans on a training and test set
body: <p>I am using dplyr and broom to compute kmeans for my data. My data contains a test and training set of X and Y coordinates and are grouped by a some parameter value (lambda in this case): </p>
  
  <pre><code>mds.test = data.frame()
  for(l in seq(0.1, 0.9, by=0.2)) {
    new.dist &lt;- run.distance.model(x, y, lambda=l)
    mds &lt;- preform.mds(new.dist, ndim=2)
    mds.test &lt;- rbind(mds.test, cbind(mds$space, design[,c(1,3,4,5)], lambda=rep(l, nrow(mds$space)), data="test"))
  }
  
  &gt; head(mds.test)
                          Comp1       Comp2 Transcripts Genes Timepoint Run lambda data
  7A_0_AAGCCTAGCGAC -0.06690476 -0.25519106       68125  9324     Day 0  7A    0.1 test
  7A_0_AAATGACTGGCC -0.15292848  0.04310200       28443  6746     Day 0  7A    0.1 test
  7A_0_CATCTCGTTCTA -0.12529445  0.13022908       27360  6318     Day 0  7A    0.1 test
  7A_0_ACCGGCACATTC -0.33015913  0.14647857       23038  5709     Day 0  7A    0.1 test
  7A_0_TATGTCGGAATG -0.25826098  0.05424976       22414  5878     Day 0  7A    0.1 test
  7A_0_GAAAAAGGTGAT -0.24349387  0.08071162       21907  6766     Day 0  7A    0.1 test
  </code></pre>
  
  <p>I've <code>head</code> the test dataset above but I also have one named <code>mds.train</code> which contains my training data coordinates. My ultimate goal here is to run k-means for both sets grouped by lambda, <strong>then compute the within.ss, between.ss and total.ss for the test data on the training centers</strong>. Thanks to <a href="https://cran.r-project.org/web/packages/broom/vignettes/kmeans.html" rel="noreferrer">a great resource</a> on broom, I am able to run kmeans for each lambda for the test set by simply doing the following: </p>
  
  <pre><code>test.kclusts  = mds.test %&gt;% 
    group_by(lambda) %&gt;% 
    do(kclust=kmeans(cbind(.$Comp1, .$Comp2), centers=length(unique(design$Timepoint))))
  </code></pre>
  
  <p>Then I can compute the centers of this data for each cluster within each lambda: </p>
  
  <pre><code>test.clusters = test.kclusts %&gt;% 
    group_by(lambda) %&gt;%  
    do(tidy(.$kclust[[1]])) 
  </code></pre>
  
  <p>This is where I am stuck. How do I compute the feature assignments as similarly shown on the <a href="https://cran.r-project.org/web/packages/broom/vignettes/kmeans.html" rel="noreferrer">reference page</a> (e.g. <code>kclusts %&gt;% group_by(k) %&gt;% do(augment(.$kclust[[1]], points.matrix))</code>), where my <code>points.matrix</code> is <code>mds.test</code> which is a data.frame with <code>length(unique(mds.test$lambda))</code> times as many rows as should be? And is there a way to somehow use centers from the training set to compute <code>glance()</code> statistics based off the test assignments?</p>
  
  <p>Any help would be greatly appreciated! Thank you!</p>
  
  <p>EDIT: Updating progress. I have figured out how to aggregate the test/training assignments but am still having issues trying to compute kmeans statistics from both sets (training assignments on test center and test assignments on training centers). Updated code is below:</p>
  
  <pre><code>test.kclusts  = mds.test %&gt;% group_by(lambda) %&gt;% do(kclust=kmeans(cbind(.$Comp1, .$Comp2), centers=length(unique(design$Timepoint))))
  test.clusters = test.kclusts %&gt;% group_by(lambda) %&gt;%  do(tidy(.$kclust[[1]])) 
  test.clusterings = test.kclusts %&gt;% group_by(lambda) %&gt;% do(glance(.$kclust[[1]]))
  test.assignments = left_join(test.kclusts, mds.test) %&gt;% group_by(lambda) %&gt;% do(augment(.$kclust[[1]], cbind(.$Comp1, .$Comp2)))
  
  train.kclusts  = mds.train %&gt;% group_by(lambda) %&gt;% do(kclust=kmeans(cbind(.$Comp1, .$Comp2), centers=length(unique(design$Timepoint))))
  train.clusters = train.kclusts %&gt;% group_by(lambda) %&gt;%  do(tidy(.$kclust[[1]])) 
  train.clusterings = train.kclusts %&gt;% group_by(lambda) %&gt;% do(glance(.$kclust[[1]]))
  train.assignments = left_join(train.kclusts, mds.train) %&gt;% group_by(lambda) %&gt;% do(augment(.$kclust[[1]], cbind(.$Comp1, .$Comp2)))
  
  test.assignments$data = "test"
  train.assignments$data = "train"
  merge.assignments = rbind(test.assignments, train.assignments)
  merge.assignments %&gt;% filter(., data=='test') %&gt;% group_by(lambda) ... ? 
  </code></pre>
  
  <p>Ive attached a plot below which illustrates my progress to this point. Just to reiterate, I would like to compute kmeans statistics (within sum of square, total sum of squares, and between sum of squares) for the training data centers on test assignments/coordinates (the plots which the centers look off): 
  <a href="https://i.stack.imgur.com/Pfd8x.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Pfd8x.png" alt="enter image description here"></a></p>