title: List Transformation With Lambdas in Spark
body: <p>I am attempting to take an RDD containing pairs of integer ranges, and transform it so that each pair has a third term which iterates through the possible values in the range.  Basically, I've got this:</p>
  
  <pre><code>[[1,10], [11,20], [21,30]]
  </code></pre>
  
  <p>And I'd like to end up with this:</p>
  
  <pre><code>[[1,1,10], [2,1,10], [3,1,10], [4,1,10], [5,1,10]...]
  </code></pre>
  
  <p>The file I'd like to transform is very large, which is why I'm looking to do this with PySpark rather than just Python on a local machine (I've got a way to do it locally on a CSV file, but the process takes several hours given the file's size).  So far, I've got this:</p>
  
  <pre><code>a = [[1,10], [11,20], [21,30]]
  b = sc.parallelize(a)
  c = b.map(lambda x: [range(x[0], x[1]+1), x[0], x[1]])
  c.collect()
  </code></pre>
  
  <p>Which yields:</p>
  
  <pre><code>&gt;&gt;&gt; c.collect()
  [[[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 1, 10], [[11, 12, 13, 14, 15, 16, 17, 18, 19, 20], 11, 20], [[21, 22, 23, 24, 25, 26, 27, 28, 29, 30], 21, 30]]
  </code></pre>
  
  <p>I can't figure out what the next step needs to be from here, to iterate over the expanded range, and pair each of those with the range delimiters.</p>
  
  <p>Any ideas?</p>
  
  <p><strong>EDIT 5/8/2017 3:00PM</strong></p>
  
  <p>The local Python technique that works on a CSV input is:</p>
  
  <pre><code>import csv
  import gzip
  csvfile_expanded = gzip.open('C:\output.csv', 'wb')
  ranges_expanded = csv.writer(csvfile_expanded, delimiter=',', quotechar='"')
  csvfile = open('C:\input.csv', 'rb')
  ranges = csv.reader(csvfile, delimiter=',', quotechar='"')
  for row in ranges:
      for i in range(int(row[0]),int(row[1])+1):
           ranges_expanded.writerow([i,row[0],row[1])
  </code></pre>
  
  <p>The PySpark script I'm questioning begins with the CSV file already having been loaded into HDFS and cast as an RDD.</p>