title: Merge pyspark dataframe rows based on the same ID
body: <p>I want to flatten hbase dataset. Individual has 2 types of data in other table, mail and phone. After joining them on rel_id=data_id i'm getting 2 separate records for one individual.</p>
  
  <pre><code>    |individual_id|rel_id|data_id|       mail|phone|role
      ---------------------------------------------------------
      |            1|   100|    100|a@gmail.com| null|secondary
      |            1|   200|    200|       null|  123|primary
      |            2|   300|    300|       null|  345|secondary
      |            2|   400|    400|b@gmail.com| null|primary
      ------------------------------------------------
  </code></pre>
  
  <p>Is it possible somehow to merge mail and phone columns for the same individual_id and create new columns for each data type role so i would have data like this: </p>
  
  <pre><code>    |individual_id|       mail|phone|mail_role |phone_role
      ------------------------------------------------------
      |            1|a@gmail.com|  123| secondary|primary
      |            2|b@gmail.com|  345|   primary|secondary
      -----------------------------------------------------
  </code></pre>
  
  <p>I would really appreciate any help.</p>