title: How to check for intersection of two DataFrame columns in Spark
body: <p>Using either <code>pyspark</code> or <code>sparkr</code> (preferably both), how can I get the intersection of two <code>DataFrame</code> columns? For example, in <code>sparkr</code> I have the following <code>DataFrames</code>:</p>
  
  <pre><code>newHires &lt;- data.frame(name = c("Thomas", "George", "George", "John"),
                         surname = c("Smith", "Williams", "Brown", "Taylor"))
  salesTeam &lt;- data.frame(name = c("Lucas", "Bill", "George"),
                          surname = c("Martin", "Clark", "Williams"))
  newHiresDF &lt;- createDataFrame(newHires)
  salesTeamDF &lt;- createDataFrame(salesTeam)
  
  #Intersect works for the entire DataFrames
  newSalesHire &lt;- intersect(newHiresDF, salesTeamDF)
  head(newSalesHire)
  
          name  surname
      1 George Williams
  
  #Intersect does not work for single columns
  newSalesHire &lt;- intersect(newHiresDF$name, salesTeamDF$name)
  head(newSalesHire)
  </code></pre>
  
  <blockquote>
    <p>Error in as.vector(y) : no method for coercing this S4 class to a
    vector</p>
  </blockquote>
  
  <p>How can I get <code>intersect</code> to work for single columns?</p>