title: Scala access or pass dataframe within udf
body: <p>UDF - I have to traverse through the dataframe to perform certain operation. I do understand that I cannot access the complete dataset in udf. But what is the solution to this?</p>
  
  <pre><code>def process(spark: SparkSession, df: DataFrame): DataFrame = { 
  
      def udf_func = udf((name: String, cluster_id: String) =&gt; {
      print(df.count()) &lt;-- thisline   
      cluster_id     
  })
  val newdf = df.withColumn("cluster_id", udf_func(df("name"),df("cluster_id")))
  </code></pre>
  
  <p>I need to traverse 'name' through all the names of the dataframe and get levenshtein distance, if it is within a threadshold, I update the cluster_id</p>
  
  <p>print(df.count()) --> Demo only - just to show that I cannot access df within the UDF</p>