title: How to use first and last function in pyspark?
body: <p>I used first and last functions to get first and last values of one column. But, I found the both of functions don't work as what I supposed. I referred to the <a href="https://stackoverflow.com/questions/35142216/first-value-windowing-function-in-pyspark#35142887">answer @zero323</a>, but I am still confusing with the both. the code like:</p>
  
  <pre><code>df = spark.sparkContext.parallelize([
      ("a", None), ("a", 1), ("a", -1), ("b", 3), ("b", 1)
  ]).toDF(["k", "v"])
  w = Window().partitionBy("k").orderBy('k','v')
  
  df.select(F.col("k"), F.last("v",True).over(w).alias('v')).show()
  </code></pre>
  
  <p>the result:</p>
  
  <pre><code>+---+----+
  |  k|   v|
  +---+----+
  |  b|   1|
  |  b|   3|
  |  a|null|
  |  a|  -1|
  |  a|   1|
  +---+----+
  </code></pre>
  
  <p>I supposed it should be like:</p>
  
  <pre><code>+---+----+
  |  k|   v|
  +---+----+
  |  b|   3|
  |  b|   3|
  |  a|   1|
  |  a|   1|
  |  a|   1|
  +---+----+
  </code></pre>
  
  <p>because, I showed df by operation of orderBy on 'k' and 'v':</p>
  
  <pre><code>df.orderBy('k','v').show()
      +---+----+
      |  k|   v|
      +---+----+
      |  a|null|
      |  a|  -1|
      |  a|   1|
      |  b|   1|
      |  b|   3|
      +---+----+
  </code></pre>
  
  <p>additionally, I figured out the other solution to test this kind of problems, my code like:</p>
  
  <pre><code>df.orderBy('k','v').groupBy('k').agg(F.first('v')).show()
  </code></pre>
  
  <p>I found that it was possible that its results are different after running above it every time . Was someone met the same experience like me? I hope to use the both of functions in my project, but I found those solutions are inconclusive.</p>