title: adding a unique consecutive row number to dataframe in pyspark
body: <p>I want to add the unique row number to my dataframe in pyspark and dont want to use monotonicallyIncreasingId &amp; partitionBy methods.
  I think that this question might be a duplicate of similar questions asked earlier, still looking for some advice whether I am doing it right way or not.
  following is snippet of my code:
  I have a csv file with below set of input records:</p>
  
  <pre><code>1,VIKRANT SINGH RANA    ,NOIDA   ,10000
  3,GOVIND NIMBHAL        ,DWARKA  ,92000
  2,RAGHVENDRA KUMAR GUPTA,GURGAON ,50000
  4,ABHIJAN SINHA         ,SAKET   ,65000
  5,SUPER DEVELOPER       ,USA     ,50000
  6,RAJAT TYAGI           ,UP      ,65000
  7,AJAY SHARMA           ,NOIDA   ,70000
  8,SIDDHARTH BASU        ,SAKET   ,72000
  9,ROBERT                ,GURGAON ,70000
  </code></pre>
  
  <p>and I have loaded this csv file into a dataframe.</p>
  
  <pre><code>PATH_TO_FILE="file:///u/user/vikrant/testdata/EMP_FILE.csv"
  
  emp_df = spark.read.format("com.databricks.spark.csv") \
    .option("mode", "DROPMALFORMED") \
    .option("header", "true") \
    .option("inferschema", "true") \
    .option("delimiter", ",").load(PATH_TO_FILE)
  
  +------+--------------------+--------+----------+
  |emp_id|            emp_name|emp_city|emp_salary|
  +------+--------------------+--------+----------+
  |     1|VIKRANT SINGH RAN...|NOIDA   |     10000|
  |     3|GOVIND NIMBHAL   ...|DWARKA  |     92000|
  |     2|RAGHVENDRA KUMAR ...|GURGAON |     50000|
  |     4|ABHIJAN SINHA    ...|SAKET   |     65000|
  |     5|SUPER DEVELOPER  ...|USA     |     50000|
  |     6|RAJAT TYAGI      ...|UP      |     65000|
  |     7|AJAY SHARMA      ...|NOIDA   |     70000|
  |     8|SIDDHARTH BASU   ...|SAKET   |     72000|
  |     9|ROBERT           ...|GURGAON |     70000|
  +------+--------------------+--------+----------+
  
  empRDD = emp_df.rdd.zipWithIndex()
  newRDD=empRDD.map(lambda x: (list(x[0]) + [x[1]]))
   newRDD.take(2);
  [[1, u'VIKRANT SINGH RANA    ', u'NOIDA   ', 10000, 0], [3, u'GOVIND NIMBHAL        ', u'DWARKA  ', 92000, 1]]
  </code></pre>
  
  <p>when I included the int value to my list, I have lost the dataframe schema.</p>
  
  <pre><code>newdf=newRDD.toDF(['emp_id','emp_name','emp_city','emp_salary','row_id'])
  newdf.show();
  
  +------+--------------------+--------+----------+------+
  |emp_id|            emp_name|emp_city|emp_salary|row_id|
  +------+--------------------+--------+----------+------+
  |     1|VIKRANT SINGH RAN...|NOIDA   |     10000|     0|
  |     3|GOVIND NIMBHAL   ...|DWARKA  |     92000|     1|
  |     2|RAGHVENDRA KUMAR ...|GURGAON |     50000|     2|
  |     4|ABHIJAN SINHA    ...|SAKET   |     65000|     3|
  |     5|SUPER DEVELOPER  ...|USA     |     50000|     4|
  |     6|RAJAT TYAGI      ...|UP      |     65000|     5|
  |     7|AJAY SHARMA      ...|NOIDA   |     70000|     6|
  |     8|SIDDHARTH BASU   ...|SAKET   |     72000|     7|
  |     9|ROBERT           ...|GURGAON |     70000|     8|
  +------+--------------------+--------+----------+------+
  </code></pre>
  
  <p>Am I doing it right way? or is there any better way to add or preserve the schema of dataframe in pyspark?</p>
  
  <p>Is it feasible to use zipWithIndex method to add unique consecutive row number for large size dataframe also? Can we use this row_id to re-partition the dataframe to uniformly distribute the data across the partitions?</p>