title: select and calculate a new column in a spark dataframe pyspark
body: <p>I have a spark dataframe with this format:   </p>
  
  <pre><code> opp_id__reference|oplin_status|               stage|      std_amount| std_line_amount|
      +-----------------+------------+--------------------+----------------+----------------+
      |OP-171102-67318|         Won|7 - Deliver &amp; Val...|6243.316662349|6243.31666234948|
      |OP-180910-77114|         Won|7 - Deliver &amp; Val...|5014.57880858921|5014.57880858921|
      |OP-180910-76544|         Pending|7 - Deliver &amp; Val...|5014.57880858921|5014.57880858921|
      |OP-180910-76544|         Pending|7 - Deliver &amp; Val...|5014.57880858921|5614.57880858921|
      |OP-180910-76544|         Won|7 - Deliver &amp; Val...|5014.57880858921|5994.57880858921|
  </code></pre>
  
  <p>I would like to extract the list of <code>opp_id__reference</code> that the sum of records which has <strong>oplin_status =</strong> <strong>"Pending"</strong> is bigger than <strong>std_amount</strong></p>
  
  <p>This hiw I did : </p>
  
  <pre><code># select opp_line which stage =='7 - Deliver &amp; Validate'  and oplin_status =='Pending'
  DF_BR8 = df.filter(df.stage.contains("7 - Deliver")).select('opp_id__reference', 'oplin_status', 'stage', 'std_amount', 'std_line_amount')
  
  DF_BR8_1 = DF_BR8.groupby('opp_id__reference', 'std_amount', 'oplin_status').agg({'std_line_amount': 'sum'}).withColumnRenamed('sum(std_line_amount)','sum_column')
  
  DF_res = DF_BR8_1.filter(DF_BR8_1.oplin_status.contains("Pending"))
  DF_res1 =DF_res.filter(DF_res.sum_column &lt;= 0.3*DF_BR8_1.std_amount)
  </code></pre>
  
  <p>My question is it : is it correct what i did? is there any other way more simple to do?</p>
  
  <p>Thanks</p>