title: PySpark: replace string by numerical values
body: <p>I have the following table :</p>
  
  <pre><code> +----------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+
      |  _created|  _updated|                name|         description|          indication|                name|      patents_patent|
      +----------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+
      |2005-06-13|2016-08-17|           Lepirudin|Lepirudin is iden...|For the treatment...|           Lepirudin|{"data" : [{"coun...|
      |2005-06-13|2017-04-27|           Cetuximab|Cetuximab is an e...|Cetuximab, used i...|           Cetuximab|{"data" : [{"coun...|
      |2005-06-13|2017-06-14|        Dornase alfa|Dornase alfa is a...|Used as adjunct t...|        Dornase alfa|{"data" : [{"coun...|
      |2005-06-13|2016-08-17| Denileukin diftitox|A recombinant DNA...|For treatment of ...| Denileukin diftitox|                NULL|
      |2005-06-13|2017-03-10|          Etanercept|Dimeric fusion pr...|Etanercept is ind...|          Etanercept|{"data" : [{"coun...|
      |2005-06-13|2017-07-06|         Bivalirudin|Bivalirudin is a ...|For treatment of ...|         Bivalirudin|{"data" : [{"coun...|
      |2005-06-13|2017-07-05|          Leuprolide|Leuprolide belong...|For treatment of ...|          Leuprolide|{"data" : [{"coun...|
      |2005-06-13|2017-06-16|Peginterferon alf...|Peginterferon alf...|Peginterferon alf...|Peginterferon alf...|{"data" : [{"coun...|
      |2005-06-13|2017-06-08|           Alteplase|Human tissue plas...|For management of...|           Alteplase|                NULL|
      |2005-06-13|2016-12-08|          Sermorelin|Sermorelin acetat...|For the treatment...|          Sermorelin|                NULL|
      |2005-06-13|2016-08-17|  Interferon alfa-n1|Purified, natural...|For treatment of ...|  Interferon alfa-n1|                NULL|
  </code></pre>
  
  <p>Ideally I will need to derive 2 tables :</p>
  
  <p>table_one I will filter out table where patent_patent is not NULL and replace the strings in patent-patent by 1:</p>
  
  <pre><code>+----------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+
      |  _created|  _updated|                name|         description|          indication|                name|      patents_patent|
      +----------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+
      |2005-06-13|2016-08-17|           Lepirudin|Lepirudin is iden...|For the treatment...|           Lepirudin|1|
      |2005-06-13|2017-04-27|           Cetuximab|Cetuximab is an e...|Cetuximab, used i...|           Cetuximab|1|
      |2005-06-13|2017-06-14|        Dornase alfa|Dornase alfa is a...|Used as adjunct t...|        Dornase alfa|1|
      |2005-06-13|2017-03-10|          Etanercept|Dimeric fusion pr...|Etanercept is ind...|          Etanercept|1|
      |2005-06-13|2017-07-06|         Bivalirudin|Bivalirudin is a ...|For treatment of ...|         Bivalirudin|1|
      |2005-06-13|2017-07-05|          Leuprolide|Leuprolide belong...|For treatment of ...|          Leuprolide|1|
      |2005-06-13|2017-06-16|Peginterferon alf...|Peginterferon alf...|Peginterferon alf...|Peginterferon alf...|1|
      |
  </code></pre>
  
  <p>table_two =  filter out table where patents_patent is null and replace null by 0</p>
  
  <pre><code>    +----------+----------+--------------------+--------------------+--------------------+--------------------+--------------------+
     |  _created|  _updated|                name|         description|          indication|                name|      patents_patent|
      +----------+----------+--------------------+--------------------+--------------------+--------------------+------------------
     |2005-06-13|2016-08-17| Denileukin diftitox|A recombinant DNA...|For treatment of ...| Denileukin diftitox|                0|
  
      |2005-06-13|2017-06-08|           Alteplase|Human tissue plas...|For management of...|           Alteplase|                0|
      |2005-06-13|2016-12-08|          Sermorelin|Sermorelin acetat...|For the treatment...|          Sermorelin|                0|
      |2005-06-13|2016-08-17|  Interferon alfa-n1|Purified, natural...|For treatment of ...|  Interferon alfa-n1|                0|
  </code></pre>
  
  <p>I tried this:</p>
  
  <p>I tried this :  </p>
  
  <pre><code>from pyspark.sql.functions import col, expr, when
  
  data = table.where(col("patents_patent").isNull())
  
  data = table.filter("patents_patent is not NULL")
  </code></pre>
  
  <p>The results are wrong or empty :!</p>
  
  <pre><code>root
   |-- _created: string (nullable = true)
   |-- _updated: string (nullable = true)
   |-- name: string (nullable = true)
   |-- description: string (nullable = true)
   |-- indication: string (nullable = true)
   |-- patents_patent: string (nullable = true)
  </code></pre>
  
  <p>Thanks for the help !</p>