Can datasets remove duplicated rows?
Hello,

I'm also interested in this feature.
Has there been progress on this issue?

Could we use a similar trick as above, but with a better hashing algorithm like SHA?

We could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?
**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no
Can datasets remove duplicated rows? 
 **Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no 
 Hello,

I'm also interested in this feature.
Has there been progress on this issue?

Could we use a similar trick as above, but with a better hashing algorithm like SHA?

We could also use a [bloom filter](https://en.wikipedia.org/wiki/Bloom_filter), should we care a lot about collision in this case?