title: Iterate pyspark dataframe rows and apply UDF
body: <p>I have a dataframe that looks like this:
  partitionCol orderCol valueCol</p>
  
  <pre><code>+--------------+----------+----------+
  | partitionCol | orderCol | valueCol |
  +--------------+----------+----------+
  | A            | 1        | 201      |
  | A            | 2        | 645      |
  | A            | 3        | 302      |
  | B            | 1        | 335      |
  | B            | 2        | 834      |
  +--------------+----------+----------+
  
  </code></pre>
  
  <p>I want to group by the partitionCol, then within each partition to iterate over the rows, ordered by orderCol and apply some function to calculate a new column based on the valueCol and a cached value.
  e.g.</p>
  
  <pre class="lang-py prettyprint-override"><code>def foo(col_value, cached_value):
      tmp = &lt;some value based on a condition between col_value and cached_value&gt;
      &lt;update the cached_value using some logic&gt;
      return tmp
  </code></pre>
  
  <p>I understand I need to groupby the partitionCol and apply a UDF that will operate on each chink separately, but struggling to find a good way to iterate the rows and applying the logic I described, to get a desired output of:</p>
  
  <pre><code>+--------------+----------+----------+---------------+
  | partitionCol | orderCol | valueCol | calculatedCol -
  +--------------+----------+----------+---------------+
  | A            | 1        | 201      | C1            |
  | A            | 2        | 645      | C1            |
  | A            | 3        | 302      | C2            |
  | B            | 1        | 335      | C1            |
  | B            | 2        | 834      | C2            |
  +--------------+----------+----------+---------------+
  </code></pre>