title: How to convert NULLs in a Dataframe String column to a default value using WHEN condition in pyspark?
body: <p>I have a pyspark dataframe obtained from reading a table as below.</p>
  <pre><code>from pyspark.sql import functions as F
  from pyspark.sql.functions import col
  
  data = spark.sql(&quot;select employe_header, employee_number from DB.TABLE where substring(SOME_DATECOLUMN, 1, 4) in ('2021') and employee_number is null;&quot;)
  </code></pre>
  <p>My criteria is to check if the employee_number is empty or NULL or zeroes and update it to <code>NA</code> if any of the conditions are satisfied.
  I wrote the code like below.</p>
  <pre><code>data = data.withColumn(&quot;employee_number&quot;, F.when((col(&quot;employee_number&quot;) == &quot;&quot;) | (col(&quot;employee_number&quot;).isNull()) | (col(&quot;employee_number&quot;) == &quot;0&quot;), &quot;NA&quot;))
  data.printSchema()
  </code></pre>
  <p>Schema:</p>
  <pre><code>root
   |-- employe_header: string (nullable = true)
   |-- employee_number: string (nullable = true)
  </code></pre>
  <p>But when I check the data inside dataframe, it still contains NULLs in the column: employee_number which can be seen below.
  <code>data.select('employee_number').display()</code>
  <a href="https://i.stack.imgur.com/gGEdT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gGEdT.png" alt="enter image description here" /></a></p>
  <p>Could anyone let me know what is the mistake I am doing here and how can I correct it ?</p>