title: avoiding for loop in PySpark
body: <p>I have a PySpark DataFrame similar to this:</p>
  
  <pre><code>ID | value | period
  a  |  100  |   1   
  a  |  100  |   1   
  b  |  100  |   1   
  a  |  100  |   2   
  b  |  100  |   2   
  a  |  100  |   3
  </code></pre>
  
  <p>For each period <code>(1, 2, 3)</code> I want to filter the data to where period is less than or equal to that number, then sum the value column for each ID.</p>
  
  <p>So for example, period 1 would give <code>(a:200, b:100)</code>, period 2 would give <code>(a:300, b:200)</code> and period 3 would give <code>(a:400, b:200)</code>.</p>
  
  <p>At the moment I'm doing this in a loop:</p>
  
  <pre><code>vals = [('a', 100, 1),
          ('a', 100, 1),
          ('b', 100, 1),
          ('a', 100, 2),
          ('b', 100, 2),
          ('a', 100, 3)]
  cols = ['ID', 'value', 'period']
  df = spark.createDataFrame(vals, cols)
  
  for p in (1, 2, 3):
      df_filter = df[df['period'] &lt;= p]
      results = df_filter.groupBy('ID').agg({'value':'sum'})
  </code></pre>
  
  <p>I then convert the "results" to pandas and append them into one DataFrame.</p>
  
  <p>Is there a better way of doing this without having to use a loop? (in practice I have hundreds of periods).</p>