title: Pyspark: Using a row in one df to filter and select another
body: <p>I am trying to iterate over rows in one PySpark dataframe, and use values in each row to perform operations (filter, select) on a 2nd Pyspark dataframe, then bind all results. Perhaps this is best illustrated:</p>
  <p>DF1</p>
  <pre class="lang-none prettyprint-override"><code>id   name   which_col
  1    John   col1
  2    Jane   col3
  3    Bob    col2
  4    Barb   col1
  </code></pre>
  <p>DF2</p>
  <pre class="lang-none prettyprint-override"><code>name  col1  col2  col3
  Bob   78    43    54
  Bob   23    65    34
  Bob   12    98    75
  John  45    54    54
  John  75    43    12
  Jane  24    45    21
  ...
  </code></pre>
  <p>The steps I'd like to perform for every row in DF1 are:</p>
  <ol>
  <li>Take value in 'name' and use it to filter DF2 (e.g., for row 1, filter DF2 to only &quot;John&quot; rows.)</li>
  <li>Then select the corresponding DF2 column based on the DF1 value in &quot;which_col&quot; (e.g., for John, select col1 in DF2, whereas for Jane it would be col3).</li>
  <li>Repeat for each row of DF1</li>
  <li>Bind all results together into a final DF.</li>
  </ol>