title: Compare 2 arrays and only remove the instances of values that exist in both arrays pyspark
body: <p>I have 2 array columns</p>
  <pre><code>| A                      | B                |
  | -----------------------|------------------|
  | [0.0001,0.0045,0.0031] |[0.0001,0.0031]   |
  | [2.5,1.0,0.5,3.0]      | [2.5, 0.5, 3.0]  |
  | [1.0,1.0,1.5,1.0]      | [1.0,1.0]        |
  </code></pre>
  <p>I want to create a new column that is an array of only the values that are not included in both arrays, but I want it to pay attention to instances, for example if .0001 exists in both arrays, but it exists once in one array and twice in the other, I want it to leave 1 remaining instance of .0001.</p>
  <pre><code>| A                      | B                | C               |
  | -----------------------|------------------|-----------------|
  | [0.0001,0.0045,0.0031] |[0.0001,0.0031]   |[0.0045]         |    
  | [2.5,1.0,0.5,3.0]      | [2.5, 0.5, 3.0]  |[1.0]            |
  | [1.0,1.0,1.5,1.0]      | [1.0,1.0]        |[1.0, 1.5]       |
  </code></pre>
  <p>Any ideas how to accomplish this with Pyspark? As of now I have been using df.withColumn(&quot;C&quot;, F.array_except(F.col(&quot;A&quot;), F.col(&quot;B&quot;))), but this takes away all instances of the numbers.</p>