title: How to add and update a column in dataframe in spark
body: <p>I've got a dataframe with millions of records and I want to add a new column and update it in spark using scala based on some condition as shown below :</p>
  
  <pre><code>uid       date  code
  1   20-05-2016     a
  1   21-05-2016     b
  1   22-05-2016     c
  2   21-05-2016     a
  2   23-06-2016     c
  2   24-06-2016     b
  2   24-06-2016     c
  2   25-06-2016     d
  </code></pre>
  
  <p>I want to add a column to assign subid in such a way that each time code reaches c for a particular uid, the subid is incremented by 1 for that uid. So the resultant data frame would be something like :</p>
  
  <pre><code>uid       date  code  subid
  1   20-05-2016     a      1
  1   21-05-2016     b      1
  1   22-05-2016     c      1
  2   21-05-2016     a      1
  2   23-06-2016     c      1
  2   24-06-2016     b      2
  2   24-06-2016     c      2
  2   25-06-2016     d      3
  </code></pre>
  
  <p>i.e. we start assigning each uid by 1, after we reach a c for the same uid, we increment the subid by 1 and assign 2 and so on.</p>
  
  <p>Now, the challenge here is:
  1) I need to make sure that each subset of unique uid are not divided into different nodes else it could break the logic so 
  2) I read that frame specification in window functions is not supported in hive <a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-windows.html" rel="nofollow">as shown here</a>.</p>
  
  <p>Any suggestions how to achieve this whilst ensuring that i manage to cater to the volume of the dataframe and also that a all rows of an unique uid are processed on a single node.</p>