title: spark filter (delete) rows based on values from another dataframe
body: <p>I have a 'big' dataset (<code>huge_df</code>) with >20 columns. One of the columns is an <code>id</code> field (generated with <code>pyspark.sql.functions.monotonically_increasing_id()</code>).</p>
  
  <p>Using some criteria I generate a second dataframe (<code>filter_df</code>), consisting of <code>id</code> values I want to filter later on from <code>huge_df</code>.</p>
  
  <p>Currently I am using SQL syntax to do this:</p>
  
  <pre><code>filter_df.createOrReplaceTempView('filter_view')
  huge_df = huge_df.where('id NOT IN (SELECT id FROM filter_view)')
  </code></pre>
  
  <p><strong>Question 1:</strong>
  Is there a way to do this using Python only, i.e. without the need to register the <code>TempView</code>?</p>
  
  <p><strong>Question 2:</strong>
  Is there a completely different way to accomplish the same thing?</p>