title: pyspark -- best way to sum values in column of type Array(Integer())
body: <p>Lets say this is my dataframe ...</p>
  
  <pre><code>name | scores
  Dan  |  [10,5,2,12]
  Ann  |  [ 12,3,5]
  Jon  |  [ ] 
  </code></pre>
  
  <p>Desired output is something like</p>
  
  <pre><code>name | scores         | Total
  Dan  |  [10,5,2,12]   | 29
  Ann  |   [ 12,3,5]    | 20
  Jon  |  [ ]           | 0
  </code></pre>
  
  <p>I made a UDF along the lines of ....</p>
  
  <pre><code>sum_cols = udf(lambda arr: if arr == [] then 0 else __builtins__.sum(arr),IntegerType())
  
  df.withColumn('Total', sum_cols(col('scores'))).show()
  </code></pre>
  
  <p>However, I have learned that UDFs are relatively slow to pure pySpark functions.</p>
  
  <p>Any way to do code above in pySpark without a UDF ?</p>