title: How to retain the preceding updated row values in PySpark and use it in the next row calculation?
body: <p>The below condition needs to be applied on RANK and RANKA columns</p>
  <p>Input table:</p>
  <p><a href="https://i.stack.imgur.com/0Y7Sh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0Y7Sh.png" alt="input" /></a></p>
  <p><strong>Condition for RANK column:</strong></p>
  <ul>
  <li><p>IF RANK == 0 : then RANK= previous RANK value + 1 ;</p>
  </li>
  <li><p>else :            RANK=RANK</p>
  </li>
  </ul>
  <p><strong>Condition for RANKA column:</strong></p>
  <ul>
  <li>IF RANKA == 0 : then RANKA= previous RANKA value + current row Salary
  value;</li>
  <li>else :         RANKA=RANKA</li>
  </ul>
  <p><strong>Below is a piece of code that I tried.</strong></p>
  <p>I have created dummy columns named <em><strong>RANK_new</strong></em> and <em><strong>RANKA_new</strong></em> for storing the desired outputs of RANK and RANKA columns after applying conditions.
  And then once I get the correct values I can replace the RANK and RANKA column with those dummy columns.</p>
  <pre class="lang-py prettyprint-override"><code># importing necessary libraries
  import pyspark
  from pyspark.sql import SparkSession
  from pyspark.sql.window import Window
  from pyspark.sql.functions import when, col
  # function to create new SparkSession
  from pyspark.sql.functions import lit
  from pyspark.sql.functions import lag,lead
  def create_session():
      spk = SparkSession.builder \
          .master(&quot;local&quot;) \
          .appName(&quot;employee_profile.com&quot;) \
          .getOrCreate()
      return spk
   
   
  def create_df(spark, data, schema):
      df1 = spark.createDataFrame(data, schema)
      return df1
   
   
  if __name__ == &quot;__main__&quot;:
   
      # calling function to create SparkSession
      spark = create_session()
   
      input_data = [(1, &quot;Shivansh&quot;, &quot;Data Scientist&quot;, 2,1,1,2),
                    (0, &quot;Rishabh&quot;, &quot;Software Developer&quot;, 5,2,0,3),
                    (0, &quot;Swati&quot;, &quot;Data Analyst&quot;, 10,3,10,4),
                    (1, &quot;Amar&quot;, &quot;Data Analyst&quot;, 2,4,9,0),
                    (0, &quot;Arpit&quot;, &quot;Android Developer&quot;, 3,5,0,0),
                    (0, &quot;Ranjeet&quot;, &quot;Python Developer&quot;, 4,6,0,0),
                    (0, &quot;Priyanka&quot;, &quot;Full Stack Developer&quot;,5,7,0,0)]
   
      schema = [&quot;Id&quot;, &quot;Name&quot;, &quot;Job Profile&quot;, &quot;Salary&quot;,'hi','RANK','RANKA'] 
      # calling function to create dataframe
      dff = create_df(spark, input_data, schema) 
      
      # Below 3 lines for RANK
      df1=dff.repartition(1)  
      df2 = df1.withColumn('RANK_new', when(col('RANK') == 0,lag(col('RANK')+lit(1)).over(Window.orderBy(col('hi')))).otherwise(col('RANK')))
      df2 = df2.withColumn('RANK_new', when((col('RANK') == 0) &amp; (lag(col('RANK')).over(Window.orderBy(col('hi'))) == 0) ,lag(col('RANK_new')+lit(1)).over(Window.orderBy(col('hi')))).otherwise(col('RANK_new')))
      
      #Below line for RANKA
      df2=df2.withColumn('RANKA_new', when(col('RANKA') == 0, lag(col(&quot;RANKA&quot;)).over(Window.orderBy(&quot;hi&quot;))+col(&quot;Salary&quot;)).otherwise(col('RANKA')))
      df2.show()
  </code></pre>
  <p>The issue with this code is that <strong>the lag function is not taking the updated values of the previous rows.</strong>
  <strong>This can be done with a for loop but since my data is so huge, I need a solution without for loop.</strong></p>
  <p>Below is the desired output:</p>
  <p><a href="https://i.stack.imgur.com/r7eRN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/r7eRN.png" alt="Desired output" /></a></p>
  <p>Below is a summarized picture to show the Output I got and the desired output.</p>
  <p><strong>RANK_new, RANKA_new</strong> --&gt; These are the output I got for RANK and RANKA columns after I applied the above code</p>
  <p><strong>RANK_desired, RANKA-desired</strong> ---&gt; <strong>This is what is expected to be produced</strong>.</p>
  <p><a href="https://i.stack.imgur.com/uo3Xk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uo3Xk.png" alt="summarized " /></a></p>