title: Need to read and then remove duplicates from multiple CSV file in Spark scala
body: <p>I am new to this space,I have multiple partitioned CSV files having duplicates records. I want to read the CSV file in Spark Scala code and remove duplicates as well while reading.</p>
  
  <p>I have tried <code>dropDuplicate()</code> and <code>read.format("csv")</code> with load option.</p>
  
  <pre><code>var df1 = thesparksession.read.format("csv").option("delimiter","|").option("header",true).load("path/../../*csv)
  .withcolumn(col1)
  df1.dropDuplicates().show()
  </code></pre>
  
  <p>if lets say csv1 has values</p>
  
  <pre><code>emp1 1000 nuu -1903.33
  emp2 1003 yuu 1874.44
  </code></pre>
  
  <p>and csv2 has</p>
  
  <pre><code>emp1 1000 nuu -1903.33
  emp4 9848 hee 1874.33
  </code></pre>
  
  <p>I need only one record with emp1 will be processed further.</p>
  
  <p>expected output :</p>
  
  <pre><code>emp1 1000 nuu -1903.33
  emp2 1003 yuu 1874.44
  emp4 9848 hee 1874.33
  </code></pre>