title: Exclude null values in column while using windows partition by column in Pyspark
body: <p>I have a data frame like below in <code>pyspark</code>. </p>
  
  <pre><code>+-----+---+---+----+
  |tests|val|asd|cnty|
  +-----+---+---+----+
  |test1|  Y|  1|null|
  |test2|  N|  2|  UK|
  | null|  Y|  1|  UK|
  |test1|  N|  2|null|
  | null|  N|  3|null|
  |test3|  N|  4| AUS|
  |test4|  Y|  5|null|
  +-----+---+---+----+
  </code></pre>
  
  <p>I want to update the <code>val</code> column when the any given <code>tests</code> has <code>val Y</code> then all <code>val's</code> of that particular <code>tests</code> should be updated to <code>Y</code>. if not then what ever values they have. I want to exclude the records where <code>tests</code> column has <code>null</code> values.</p>
  
  <p>I have done like below</p>
  
  <pre><code>from pyspark.sql import Window
  import pyspark.sql.functions as f
  
  
  df1 = df.select('tests', f.max('val').over(Window.partitionBy('tests')).alias('val'), 'asd', 'cnty')
  </code></pre>
  
  <p>I am getting result like below</p>
  
  <pre><code>+-----+---+---+----+
  |tests|val|asd|cnty|
  +-----+---+---+----+
  |test1|  Y|  1|null|
  |test1|  Y|  2|null|
  |test2|  N|  2|  UK|
  |test3|  N|  4| AUS|
  |test4|  Y|  5|null|
  | null|  Y|  1|  UK|
  | null|  Y|  3|null|
  +-----+---+---+----+
  </code></pre>
  
  <p>I want the result to be like below</p>
  
  <pre><code>+-----+---+---+----+
  |tests|val|asd|cnty|
  +-----+---+---+----+
  |test1|  Y|  1|null|
  |test1|  Y|  2|null|
  |test2|  N|  2|  UK|
  |test3|  N|  4| AUS|
  |test4|  Y|  5|null|
  | null|  Y|  1|  UK|
  | null|  N|  3|null|
  +-----+---+---+----+
  </code></pre>