I am using Hadoop+ELK stack to build a analytic stack.I am trying to refresh the index on daily basis.

I am consuming data from a third party which is in CSV format.I have no control over the input data i.e. I cannot ask to change the schema for the CSV file.

The problem is there is no unique id in the CSV records or even combining columns to make a unique id will also not work.So,while refreshing Elasticsearch adds duplicate data to the index.

So,if day 1 data is like 

```
Product1,Language1,Date1,$1
Product2,Language2,Date1,$12

```

Day2 data becomes

```
Product1,Language1,Date1,$1
Product2,Language2,Date1,$12
Product1,Language1,Date1,$1
Product2,Language2,Date1,$12
Product3,Language1,Date2,$5(new record added on day2)

```

Is there any good way to handle this in ELK.I am using Logstash to consume the csv files.