title: Spark SQL distinct and Scala distinct give different counts, in spite of same data input data
body: <blockquote>
    <p>val apple1=spark.sql("select count(distinct *) from apple1 where data1_ts =201804").show</p>
  </blockquote>
  
  <p>10871344</p>
  
  <blockquote>
    <p>apple1.filter(col("data1_ts") === "201804").distinct.count</p>
  </blockquote>
  
  <p>20573671</p>
  
  <p>any idea both are using distinct on a table in s3,and selecting a particular directory, but gives different values?</p>