Can datasets remove duplicated rows?
An approach that works assuming you can hold the all the unique document hashes in memory:
```python
from datasets import load_dataset

def get_hash(example):
    """Get hash of content field."""
    return {"hash": hash(example["content"])} # can use any hashing function here
    
def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False

ds = load_dataset("some_dataset")
ds = ds.map(get_hash)
uniques = set(ds.unique("hash"))
ds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})
```
If the `uniques` could be stored in arrow then no additional memory would used at all but I don't know if this is possible.

**Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no
Can datasets remove duplicated rows? 
 **Is your feature request related to a problem? Please describe.**
i find myself more and more relying on datasets just to do all the preprocessing. One thing however, for removing duplicated rows, I couldn't find out how and am always converting datasets to pandas to do that..


**Describe the solution you'd like**
have a functionality of " remove duplicated rows"

**Describe alternatives you've considered**
convert dataset to pandas, remove duplicate, and convert back...


**Additional context**
no 
 An approach that works assuming you can hold the all the unique document hashes in memory:
```python
from datasets import load_dataset

def get_hash(example):
    """Get hash of content field."""
    return {"hash": hash(example["content"])} # can use any hashing function here
    
def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:
        return False

ds = load_dataset("some_dataset")
ds = ds.map(get_hash)
uniques = set(ds.unique("hash"))
ds_filter = ds.filter(check_uniques, fn_kwargs={"uniques": uniques})
```
If the `uniques` could be stored in arrow then no additional memory would used at all but I don't know if this is possible.
