title: Subtract 2 pyspark dataframes based on column
body: <p>I have 2 pyspark dataframes,</p>
  
  <pre><code>i
  +---+-----+
  | ID|COL_A|
  +---+-----+
  |  1|  123|
  |  2|  456|
  |  3|  111|
  |  4|  678|
  +---+-----+
  j
  +----+-----+
  |ID_B|COL_B|
  +----+-----+
  |   2|  456|
  |   3|  111|
  |   4|  876|
  +----+-----+
  </code></pre>
  
  <p><strong>I'm trying to subtract <code>i</code> from <code>j</code> based on values of a particular column i.e., values present in <code>COL_A</code> of <code>i</code> should not be present in <code>COL_B</code> of <code>j</code>.</strong> </p>
  
  <p>Expected output should be,</p>
  
  <pre><code>diff
  +---+-----+
  | ID|COL_A|
  +---+-----+
  |  1|  123|
  |  4|  678|
  +---+-----+
  </code></pre>
  
  <p>This is my code,</p>
  
  <pre><code>common = i.join(j.withColumnRenamed('COL_B', 'COL_A'), ['COL_A'], 'leftsemi')
  diff = i.subtract(common)
  diff.show()
  </code></pre>
  
  <p>But the output is coming wrong,</p>
  
  <pre><code>diff
  +---+-----+
  | ID|COL_A|
  +---+-----+
  |  2|  456|
  |  1|  123|
  |  4|  678|
  |  3|  111|
  +---+-----+
  </code></pre>
  
  <p>Am I doing something wrong here? Thanks in advance.</p>