title: Deduplication on Amazon Athena
body: <p>We have streaming applications storing data on S3. The S3 partitions might have duplicated records. We query the data in S3 through Athena. </p>
  
  <p>Is there a way to remove duplicates from S3 files so that we don't get them while querying from Athena?</p>