title: Columns value comparison in Spark data frame
body: <p>I have a data frame which contains huge number of records. In that DF a record can be repeated multiple times and every time when it got updated the last updated field will have the date on which the record modified. </p>
  
  <p>We have a group of columns on which we want to compare the rows of similar ids. During this comparison we want to capture what are the fields/columns has changed from previous record to current record and capture that in a "updated_columns" column of the updated record. Compare this second record to third record and identify the updated columns and capture that in "updated_columns"  field of third record, continue the same till the last record of that id and do the same thing for each and every id which has more than one entry.</p>
  
  <p>Initially we grouped the columns and created a hash out of that group of columns and compare against hash values of next row,this way it is helping me to identify records which has updates, but want the columns which got updated.</p>
  
  <p>Here I am sharing some data, which is expected outcome and that is how the final data should like look post adding updated columns (here I can say, use columns Col1, Col2, Col3, col4 and Col5 for comparison between two rows):</p>
  
  <p><a href="https://i.stack.imgur.com/lazUD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/lazUD.png" alt="enter image description here"></a></p>
  
  <p>Want to do this in a efficient way. Any one tried some thing like this.</p>
  
  <p>Looking for a help!</p>
  
  <p>~Krish.</p>