title: Pyspark: apply a function to matching partitions of multiple dataframes
body: <p>I have two Spark DataFrames, with matching pairs of partitions. I'd like to send each pair of partitions to a different executor and perform a Python function on them.</p>
  
  <p>If I just had one DataFrame, I could use <code>mapPartitions</code>, but then I couldn't access the other one. I've tried broadcasting the second one, but it's too large to fit on my executors without partitioning.</p>
  
  <p>I think that I probably need to use a function from <a href="https://spark.apache.org/docs/0.7.3/api/pyspark/pyspark.rdd.RDD-class.html" rel="nofollow noreferrer">pyspark.rdd.RDD</a>, but I'm not sure which one. Several options, like <code>groupByKey</code> and <code>cogroup</code> seem like they could be relevant, but I don't understand how they could be applied to a DataFrame object.</p>
  
  <p>Thanks in advance for your help! </p>
  
  <p>Here's my input data (each color is a partition):</p>
  
  <p><a href="https://i.stack.imgur.com/yOJ03.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yOJ03.png" alt="input data"></a></p>
  
  <p>And here's what I'd like to do:</p>
  
  <p><a href="https://i.stack.imgur.com/IHMKk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IHMKk.png" alt="mapping over matching pairs of partitions"></a></p>