title: Auto - Incrementing pyspark dataframe column values
body: <p>I am trying to generate an additional column in a dataframe with auto incrementing values based on the global value.However all the rows are generated with the same value and the value is not incrementing.</p>
  
  <p>Here is the code</p>
  
  <pre><code>def autoIncrement():
      global rec
      if (rec == 0) : rec = 1 
      else : rec = rec + 1
      return int(rec)
  
  rec=14
  </code></pre>
  
  <p>UDF</p>
  
  <pre><code>autoIncrementUDF = udf(autoIncrement,  IntegerType())
  
  
  df1 = hiveContext.sql("select id,name,location,state,datetime,zipcode from demo.target")
  
  df1.withColumn("id2", autoIncrementUDF()).show()
  </code></pre>
  
  <p>Here is the result df</p>
  
  <pre><code>+---+------+--------+----------+-------------------+-------+---+
  | id|  name|location|     state|           datetime|zipcode|id2|
  +---+------+--------+----------+-------------------+-------+---+
  | 20|pankaj| Chennai| TamilNadu|2018-03-26 11:00:00|   NULL| 15|
  | 10|geetha| Newyork|New Jersey|2018-03-27 10:00:00|   NULL| 15|
  | 25| pawan| Chennai| TamilNadu|2018-03-27 11:25:00|   NULL| 15|
  | 30|Manish| Gurgoan|   Gujarat|2018-03-27 11:00:00|   NULL| 15|
  +---+------+--------+----------+-------------------+-------+---+
  </code></pre>
  
  <p>But i am expecting the below result</p>
  
  <pre><code>+---+------+--------+----------+-------------------+-------+---+
  | id|  name|location|     state|           datetime|zipcode|id2|
  +---+------+--------+----------+-------------------+-------+---+
  | 20|pankaj| Chennai| TamilNadu|2018-03-26 11:00:00|   NULL| 15|
  | 10|geetha| Newyork|New Jersey|2018-03-27 10:00:00|   NULL| 16|
  | 25| pawan| Chennai| TamilNadu|2018-03-27 11:25:00|   NULL| 17|
  | 30|Manish| Gurgoan|   Gujarat|2018-03-27 11:00:00|   NULL| 18|
  +---+------+--------+----------+-------------------+-------+---+
  </code></pre>
  
  <p>Any help is appreciated.</p>