title: Spark - How to update value using data frame Scala
body: <p>I have two files with following structure</p>
  
  <h2>File 1</h2>
  
  <pre><code>gnk_id, matchId, timestamp
  </code></pre>
  
  <h2>File 2</h2>
  
  <pre><code>gnk_matchid, matchid
  </code></pre>
  
  <p>I want to update value of <code>gnk_id</code> in file 1 with value of <code>matchid</code> in file 2 if <code>file1.gnk_id = file2.gnk_machid</code>.</p>
  
  <p>For this I created two data frame in Spark. I was wondering whether we can update values in Spark? If not, is there any workaround for this which will provide updated final file?</p>
  
  <p>UPDATE</p>
  
  <p>I did something like this</p>
  
  <pre class="lang-scala prettyprint-override"><code>case class GnkMatchId(gnk: String, gnk_matchid: String)
  case class MatchGroup(gnkid: String, matchid: String, ts: String)
  
  val gnkmatchidRDD = sc.textFile("000000000001").map(_.split(',')).map(x =&gt; (x(0),x(1)) )
  
  val gnkmatchidDF = gnkmatchidRDD.map( x =&gt; GnkMatchId(x._1,x._2) ).toDF()
  
  val matchGroupMr = sc.textFile("part-00000").map(_.split(',')).map(x =&gt; (x(0),x(1),x(2)) ).map( f =&gt; MatchGroup(f._1,f._2,f._3.toString) ).toDF()
  
  val matchgrp_joinDF = matchGroupMr.join(gnkmatchidDF,matchGroupMr("gnkid") === gnkmatchidDF("gnk_matchid"),"left_outer")
  
  matchgrp_joinDF.map(x =&gt; if(x.getAs[String]("gnk_matchid").length != 0 ) {MatchGroup(x.getAs[String]("gnk_matchid"), x.getAs[String]("matchid"),x.getAs[String]("ts"))} else {MatchGroup(x.getAs[String]("gnkid"), x.getAs[String]("matchid"),x.getAs[String]("ts"))}).toDF().show()
  </code></pre>
  
  <p>But at last step it's failing for NULLpointerEXception</p>