title: spark dataframe calculated column
body: <p>I am learning spark (scala) and i am creating a dataframe with derived columns. I am trying to figure out the best practices.</p>
  
  <p>my use case has 2 derived columns which look for another columns for value - 
  for example -</p>
  
  <pre><code>if (col22 = "USD") then col1 = "US" elseif (col22 = "CDN" the col1 = "CA" else null)
  </code></pre>
  
  <p>another use case is </p>
  
  <pre><code>if(col23 = "us" &amp;&amp; col100 = "abc") then col2 = "10" else if (col23 = "us" &amp;&amp; col100 = "bacd" &amp;&amp; col99 is null then col2 = 11 else null)
  </code></pre>
  
  <p>Question - i have written UDF functions for the above mentioned calculations. I am wondering is there a better way of doing this? is writing a udf function is the best practice. i will be using these functions only once in my code.</p>
  
  <p>My Scala code - </p>
  
  <pre><code>def udf1 = udf((col22: String){  (col22) match {
      case col22 if (col22 == "USD") =&gt; "US"
      case col22 if (col22 == "CDN") =&gt; "CA"
      case _ =&gt; null  } })
  
  val df1= df.select($"col1", $"col2", udf1($"col22").as("newcol"), udf2($"col23", $"col100").as(newcol2))
  </code></pre>