title: Get first non-null values in group by (Spark 1.6)
body: <p>How can I get the first non-null values from a group by? I tried using <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.first" rel="noreferrer">first</a> with <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.coalesce" rel="noreferrer">coalesce</a> <code>F.first(F.coalesce("code"))</code> but I don't get the desired behavior (I seem to get the first row).</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.sql import SQLContext
  from pyspark.sql import functions as F
  
  sc = SparkContext("local")
  
  sqlContext = SQLContext(sc)
  
  df = sqlContext.createDataFrame([
      ("a", None, None),
      ("a", "code1", None),
      ("a", "code2", "name2"),
  ], ["id", "code", "name"])
  </code></pre>
  
  <p>I tried:</p>
  
  <pre><code>(df
    .groupby("id")
    .agg(F.first(F.coalesce("code")),
         F.first(F.coalesce("name")))
    .collect())
  </code></pre>
  
  <p>DESIRED OUTPUT</p>
  
  <pre><code>[Row(id='a', code='code1', name='name2')]
  </code></pre>