title: Full outer join in pyspark data frames
body: <p>I have created two data frames in <code>pyspark</code> like below. In these <code>data frames</code> I have column <code>id</code>. I want to perform a <code>full outer join</code> on these two data frames.</p>
  
  <pre><code>valuesA = [('Pirate',1),('Monkey',2),('Ninja',3),('Spaghetti',4)]
  a = sqlContext.createDataFrame(valuesA,['name','id'])
  
  a.show()
  +---------+---+
  |     name| id|
  +---------+---+
  |   Pirate|  1|
  |   Monkey|  2|
  |    Ninja|  3|
  |Spaghetti|  4|
  +---------+---+
  
  
  valuesB = [('dave',1),('Thor',2),('face',3), ('test',5)]
  b = sqlContext.createDataFrame(valuesB,['Movie','id'])
  
  b.show()
  +-----+---+
  |Movie| id|
  +-----+---+
  | dave|  1|
  | Thor|  2|
  | face|  3|
  | test|  5|
  +-----+---+
  
  
  full_outer_join = a.join(b, a.id == b.id,how='full')
  full_outer_join.show()
  
  +---------+----+-----+----+
  |     name|  id|Movie|  id|
  +---------+----+-----+----+
  |   Pirate|   1| dave|   1|
  |   Monkey|   2| Thor|   2|
  |    Ninja|   3| face|   3|
  |Spaghetti|   4| null|null|
  |     null|null| test|   5|
  +---------+----+-----+----+
  </code></pre>
  
  <p>I want to have a result like below when I do a <code>full_outer_join</code></p>
  
  <pre><code>+---------+-----+----+
  |     name|Movie|  id|
  +---------+-----+----+
  |   Pirate| dave|   1|
  |   Monkey| Thor|   2|
  |    Ninja| face|   3|
  |Spaghetti| null|   4|
  |     null| test|   5|
  +---------+-----+----+
  </code></pre>
  
  <p>I have done like below but getting some different result</p>
  
  <pre><code>full_outer_join = a.join(b, a.id == b.id,how='full').select(a.id, a.name, b.Movie)
  full_outer_join.show()
  +---------+----+-----+
  |     name|  id|Movie|
  +---------+----+-----+
  |   Pirate|   1| dave|
  |   Monkey|   2| Thor|
  |    Ninja|   3| face|
  |Spaghetti|   4| null|
  |     null|null| test|
  +---------+----+-----+
  </code></pre>
  
  <p>As you can see that I am missing <code>Id</code> <code>5</code> in  my <code>result data frame</code>.</p>
  
  <p>How can I achieve what I want?</p>