How to remove both duplicated values from two csv files in apache spark?
thx for @himanshulllTian's great answer. And I want so say some more. If you have several columns in your file; then you just want to remove record based on the key column. Also, I don't know whether your csv files have same schema.  Here is one way to deal with this situation. Let me borrow the example from himanshulllTian.
First, Let's find the records that share some key: val dupKey = df1.join(df2, "key").select("key"). Then we can find the part we hope to remove in each dataframe: val rmDF1 = df1.join(dupKey, "key"). Finally, same except action: val newDF1 = df1.except(rmDF). 
This maybe trivial. But work. Hope this help.
Newbie to apache spark. What I want to do is to remove both the duplicated keys from two csv files. I have tried dropDuplicates() and distinct() but all the do is remove one value. For example if key = 1010 appears in both the csv files, I want both of them gone. How do I do this?