title: How to find standard deviation in a column in a RDD in PySpark
body: <p>I have an <code>RDD</code> and I want to find the <code>standard deviation</code> in the data that is one of the columns of <code>RDD</code>. My current code is:</p>
  
  <pre><code>def extract(line):
      # line[11] is the column in which I want to find standard deviation
      return (line[1],line[2],line[5],line[6],line[8],line[10],line[11])
  
  inputfile1 = sc.textFile('file1.csv').zipWithIndex().filter(lambda (line,rownum): rownum&gt;=0).map(lambda (line, rownum): line)
  
  
  data = (inputfile1
      .map(lambda line: line.split(";"))
      .filter(lambda line: len(line) &gt;1 )
      .map(extract)) # Map to tuples
  </code></pre>
  
  <p><code>data</code> is the <code>RDD</code> in which I the last column (column 6) has the values among which I want to find the <code>standard deviation</code>. How can I find it?</p>
  
  <p><strong>UPDATE</strong>: My current code:</p>
  
  <pre><code>def extract(line):
      # last column is numeric but in string format
      return ((float(line[-1])))
  
  
  input = sc.textFile('file1.csv').zipWithIndex().filter(lambda (line,rownum): rownum&gt;=0).map(lambda (line, rownum): line)
  
  Data = (input
      .map(lambda line: line.split(";"))
      .filter(lambda line: len(line) &gt;1 )
      .map(extract)) # Map to tuples
  
   row = Row("val") 
   df = Data.map(row).toDF()
   df.map(lambda r: r.x).stdev()
  </code></pre>
  
  <p>When I run this I get error as: <code>AttributeError: x</code> at <code>df.map(lambda r: r.x).stdev()</code>. <strong>NOTE:</strong> Some of values in my data are negative</p>