title: spark conditional replacement of values
body: <p>For pandas I have a code snippet like this:</p>
  
  <pre><code>def setUnknownCatValueConditional(df, conditionCol, condition, colToSet, _valueToSet='KEINE'):
      df.loc[(df[conditionCol] == condition) &amp; (df[colToSet].isnull()), colToSet] = _valueToSet
  </code></pre>
  
  <p>which conditionally will replace values in a data frame.</p>
  
  <p>Trying to port this functionality to spark </p>
  
  <pre><code>df.withColumn("A", when($"A" === "x" and $"B" isNull, "replacement")).show
  </code></pre>
  
  <p>Did not work out for me </p>
  
  <pre><code>df.withColumn("A", when($"A" === "x" and $"B" isNull, "replacement")).show
  warning: there was one feature warning; re-run with -feature for details
  org.apache.spark.sql.AnalysisException: cannot resolve '((`A` = 'x') AND `B`)' due to data type mismatch: differing types in '((`A` = 'X') AND `B`)' (boolean and string).;;
  </code></pre>
  
  <p>even though df.printSchema returns a string for A and b</p>
  
  <p>What is wrong here?</p>
  
  <h1>edit</h1>
  
  <p>A minimal example:</p>
  
  <pre><code>import java.sql.{ Date, Timestamp }
  case class FooBar(foo:Date, bar:String)
  val myDf = Seq(("2016-01-01","first"),("2016-01-02","second"),("2016-wrongFormat","noValidFormat"), ("2016-01-04","lastAssumingSameDate"))
           .toDF("foo","bar")
           .withColumn("foo", 'foo.cast("Date"))
           .as[FooBar]
  
  myDf.printSchema
  root
   |-- foo: date (nullable = true)
   |-- bar: string (nullable = true)
  
  
  scala&gt; myDf.show
  +----------+--------------------+
  |       foo|                 bar|
  +----------+--------------------+
  |2016-01-01|               first|
  |2016-01-02|              second|
  |      null|       noValidFormat|
  |2016-01-04|lastAssumingSameDate|
  +----------+--------------------+
  
  myDf.withColumn("foo", when($"bar" === "noValidFormat" and $"foo" isNull, "noValue")).show
  </code></pre>
  
  <p>And the expected output</p>
  
  <pre><code>+----------+--------------------+
  |       foo|                 bar|
  +----------+--------------------+
  |2016-01-01|               first|
  |2016-01-02|              second|
  | "noValue"|       noValidFormat|
  |2016-01-04|lastAssumingSameDate|
  +----------+--------------------+
  </code></pre>
  
  <h1>edit2</h1>
  
  <p>in case chaining of conditions is required</p>
  
  <pre><code>df
      .withColumn("A",
        when(
          (($"B" === "x") and ($"B" isNull)) or
          (($"B" === "y") and ($"B" isNull)), "replacement") 
  </code></pre>
  
  <p>should work</p>