title: Joining in PySpark gives unexpected results
body: <p>
  I'm using Spark version 2.0.0 with python version 3.5. I have a parent table with two columns: GroupID and ID. I need to break the data into 3 subsets so I am using stratified sampling.
  </p>
  
  <p>
  Once I obtain my subset (sample A) I perform a left join and look for sample A IDs are null. The expected output is a subset of the parent table which does not contain any IDs from sample A.
  </p>
  
  <pre><code>spark = SparkSession.builder.enableHiveSupport().appName("myApp").getOrCreate()
  
  data = [(0, 100), (0, 101), (0, 102), (0, 103), (0, 1000), (1, 104), (1,105), (1, 106), (1, 107), (1, 1007)]
  df = spark.createDataFrame(data, ['group', 'id'])
  
  ## Stratified Sampling
  fractions = dict((int(item), 0.45) for item in np.arange(0,(2)))
  sampled_for_A = df.sampleBy("group", fractions, seed=7)
  sampled_for_A.orderBy("id").show()
  
  ## Version 1
  
  sampled_for_BC = df.join(sampled_for_A, df.id == sampled_for_A.id, "left_outer").select(df.group, df.id, sampled_for_A.group.alias("groupA"), sampled_for_A.id.alias("id_A"))
  sampled_for_BC.where(sampled_for_BC.id_A.isNull()).show()
  </code></pre>
  
  <p>
  In version 1 the BC table is populated, but the values from the parent table are copied over.
  </p>
  
  <pre><code>Group A
  +-----+----+
  |group|  id|
  +-----+----+
  |    0| 101|
  |    0| 102|
  |    1| 104|
  |    1| 105|
  |    1| 107|
  |    0|1000|
  +-----+----+
  
  BC
  +-----+----+------+----+
  |group|  id|groupA|id_A|
  +-----+----+------+----+
  |    1| 107|     1| 107|
  |    0| 103|     0| 103|
  |    1| 104|     1| 104|
  |    0|1000|     0|1000|
  |    1| 106|     1| 106|
  |    0| 100|     0| 100|
  |    1| 105|     1| 105|
  |    1|1007|     1|1007|
  |    0| 101|     0| 101|
  |    0| 102|     0| 102|
  +-----+----+------+----+
  </code></pre>
  
  <p>Version 2 createOrReplaceTempView and perform LEFT JOIN with sql query returns expected results</p>
  
  <pre><code>df.createOrReplaceTempView("parentTable")
  sampled_for_A.createOrReplaceTempView("groupA")
  
  subset_BC = spark.sql(''' 
  SELECT a.group,
     a.id,
     b.group AS group_A,
     b.id AS id_A
  FROM parentTable a
  LEFT JOIN groupA b
  ON a.id = b.id
  WHERE b.id IS NULL
  ''').show()
  </code></pre>
  
  <p>As expected:</p>
  
  <pre><code>BC
  +-----+----+-------+----+
  |group|  id|group_A|id_A|
  +-----+----+-------+----+
  |    0| 103|   null|null|
  |    1| 106|   null|null|
  |    0| 100|   null|null|
  |    1|1007|   null|null|
  +-----+----+-------+----+
  </code></pre>
  
  <p>What is it about the where clause in version 1 that I am missing? (Also my data has 100 million rows and even the sql left join performed in version 2 would behave strangely. I've also tried to repartition on group column.) Any insight is much appreciated.</p>