title: How to create a new column on pyspark under condition?
body: <p>I have the following <code>data.frame</code> in <code>spark</code></p>
  
  <pre><code>import findspark
  findspark.init()
  
  from pyspark.sql import SparkSession
  
  spark = SparkSession.builder.getOrCreate()
  sc = spark.sparkContext
  from pyspark.sql import functions as sf
  from pyspark.sql.functions import col, when, lit
  
  ddf = spark.createDataFrame([[None, 'Michael',2],
                               [30, 'Andy',3],
                               [19, 'Justin',4],
                               [30, 'James Dr No From Russia with Love Bond',6]],
                              schema=['age', 'name','weights'])
  ddf.show()
  </code></pre>
  
  <p>In this trivial example I would like to create two columns: One with the <code>weighted.mean</code> of the <code>age</code> if <code>age&gt;29</code> (with name <code>weighted_age</code>) and the other the <code>age^2</code> if <code>age&lt;=29</code> (with the name <code>age_squared</code>)</p>