title: NullPointerException when using UDF in Spark
body: <p>I have a DataFrame in Spark such as this one:</p>
  
  <pre><code>var df = List(
    (1,"{NUM.0002}*{NUM.0003}"),
    (2,"{NUM.0004}+{NUM.0003}"),
    (3,"END(6)"),
    (4,"END(4)")
  ).toDF("CODE", "VALUE")
  
  +----+---------------------+
  |CODE|                VALUE|
  +----+---------------------+
  |   1|{NUM.0002}*{NUM.0003}|
  |   2|{NUM.0004}+{NUM.0003}|
  |   3|               END(6)|
  |   4|               END(4)|
  +----+---------------------+
  </code></pre>
  
  <p>My task is to iterate through the VALUE column and do the following: check if there is a substring such as {NUM.XXXX}, get the XXXX number, get the row where $"CODE" === XXXX, and replace the {NUM.XXX} substring with the VALUE string in that row.</p>
  
  <p>I would like the dataframe to look like this in the end:</p>
  
  <pre><code>+----+--------------------+
  |CODE|               VALUE|
  +----+--------------------+
  |   1|END(4)+END(6)*END(6)|
  |   2|       END(4)+END(6)|
  |   3|              END(6)|
  |   4|              END(4)|
  +----+--------------------+
  </code></pre>
  
  <p>This is the best I've come up with:</p>
  
  <pre><code>val process = udf((ln: String) =&gt; {
    var newln = ln
    while(newln contains "{NUM."){
      var num = newln.slice(newln.indexOf("{")+5, newln.indexOf("}")).toInt 
      var new_value = df.where($"CODE" === num).head.getAs[String](1)
      newln = newln.replace(newln.slice(newln.indexOf("{"),newln.indexOf("}")+1), new_value)
    }
    newln
  })
  
  var df2 = df.withColumn("VALUE", when('VALUE contains "{NUM.",process('VALUE)).otherwise('VALUE))
  </code></pre>
  
  <p>Unfortunately, I get a NullPointerException when I try to filter/select/save <code>df2</code>, and no error when I just show <code>df2</code>. I believe the error appears when I access the DataFrame <code>df</code> within the UDF, but I need to access it every iteration, so I can't pass it as an input. Also, I've tried saving a copy of <code>df</code> inside the UDF but I don't know how to do that. What can I do here? </p>
  
  <p>Any suggestions to improve the algorithm are very welcome! Thanks!</p>