title: how to add Row id in pySpark dataframes
body: <p>I have a csv file; which i convert to DataFrame(df) in pyspark; after some transformation; I want to add a column in df; which should be simple row id (starting from 0 or 1 to N). </p>
  
  <p>I converted df in rdd and use "zipwithindex". I converted resulting rdd back to df. this approach works but it generated 250k tasks and takes a lot of time in execution. I was wondering if there is other way to do it which takes less runtime. </p>
  
  <p>following is snippet of my code; the csv file I am processing is BIG; contains billions of rows.</p>
  
  <pre><code>debug_csv_rdd = (sc.textFile("debug.csv")
    .filter(lambda x: x.find('header') == -1)
    .map(lambda x : x.replace("NULL","0")).map(lambda p: p.split(','))
    .map(lambda x:Row(c1=int(x[0]),c2=int(x[1]),c3=int(x[2]),c4=int(x[3]))))
  
  debug_csv_df = sqlContext.createDataFrame(debug_csv_rdd)
  debug_csv_df.registerTempTable("debug_csv_table")
  sqlContext.cacheTable("debug_csv_table")
  
  r0 = sqlContext.sql("SELECT c2 FROM debug_csv_table WHERE c1 = 'str'")
  r0.registerTempTable("r0_table")
  
  r0_1 = (r0.flatMap(lambda x:x)
      .zipWithIndex()
      .map(lambda x: Row(c1=x[0],id=int(x[1]))))
  
  r0_df=sqlContext.createDataFrame(r0_2)
  r0_df.show(10) 
  </code></pre>