title: Azure Databricks Scala : How to replace rows following a respective hirarchy
body: <p>Having in mind the following dataset:</p>
  <p><a href="https://i.stack.imgur.com/P0YzA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P0YzA.png" alt="enter image description here" /></a></p>
  <p>I would like to obtain</p>
  <p><a href="https://i.stack.imgur.com/qVgP5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qVgP5.png" alt="enter image description here" /></a></p>
  <p>As you can see, basically the idea is to follow the path indicated by column ACTUAL_ID until it is null (if it wasn't already)</p>
  <p>I tried to use a udf where I was passing the full initial Dataframe and the recursively would find what I want but it seems it is not possible to pass Dataframes to UDFs. I also looked into replacing a value of a row, but it seems that is not possible.</p>
  <p>My latest attempt:</p>
  <pre><code>def calculateLatestImdate(df: DataFrame, lookupId: String) : String = {
    var foundId = df.filter($&quot;ID&quot; === lookupId).select($&quot;ACTUAL_ID&quot;).first.getAs[String](&quot;ID&quot;);
    if (foundId == &quot;&quot; || foundId == null)
    {
      lookupId
    }
    else
    {
      calculateLatestImdate(df, foundId);
    }
  }
  
  val calculateLatestImdateUdf = udf((df:DataFrame, s:String) =&gt; {
    calculateLatestImdate(df,s)
  })
  
  val df = sc.parallelize(Seq((&quot;1&quot;, &quot;&quot;, &quot;A&quot;), (&quot;2&quot;, &quot;3&quot;, &quot;B&quot;), (&quot;3&quot;, &quot;6&quot;, &quot;C&quot;), (&quot;4&quot;, &quot;5&quot;, &quot;D&quot;), (&quot;5&quot;, &quot;&quot;, &quot;E&quot;), (&quot;6&quot;, &quot;&quot;, &quot;F&quot;))).toDF(&quot;ID&quot;,&quot;ACTUAL_ID&quot;, &quot;DATA&quot;)
  
  val finalDf = df.withColumn(&quot;FINAL_ID&quot;, when(isEmpty($&quot;ACTUAL_ID&quot;), $&quot;ID&quot;).otherwise(calculateLatestImdateUdf(df, $&quot;ACTUAL_ID&quot;)))
  </code></pre>