title: Spark Dataframe join where column is null
body: <p>Im trying to recreate a Left Outer Join in Spark. The join also excludes NULL values in the right table. The thought behind this is: The left table provides new records. I need to check if there's any new data and load the selection into a new dataframe.</p>
  
  <p><strong>SQL as reference (this works):</strong></p>
  
  <pre><code>SELECT 
  primary_key,
  attribute_1, 
  attribute_2
  FROM
  left_table l
  LEFT JOIN right_table r ON l.primary_key = r.foreign_key 
  WHERE r.foreign_key IS NULL
  </code></pre>
  
  <p><strong>Java Dataframe Join</strong></p>
  
  <pre><code>Dataset&lt;Row&gt; joined = left_table.alias("l").join(right_table.alias("r"), left_table.col("primary_key").equalTo(right_table.col("foreign_key")), "left_outer");
  </code></pre>
  
  <p><strong>Result</strong></p>
  
  <pre><code>+--------------------+-----------+-------+-----+
  |         primary_key|foreign_key|attr_1 |attr2|
  +--------------------+-----------+-------+-----+
  |c6d2f6791d7e6127d...|       null|   null| null|      
  |485e86b2b77ebd787...|       null|   null| null|     
  |8c7e1ac10cb1650e5...|       null|   null| null| 
  </code></pre>
  
  <p>The Join itself works but i still get the null values. The next thing i tried was adding a where clause (.where(right_table.col("foreign_key").isNull());). This just leaves me with an empty dataframe. </p>
  
  <p>How can i insert a where-clause/filter to exlude the null values?</p>