I have a spark DF as below. I need to roll up multiple rows with same ID as single row but the values should be distinct.

```
id|values
1 |hello
1 |hello Sam
1 |hello Tom
2 |hello
2 |hello Tom

```

Expected Output

```
id|values
1 |hello, Sam, Tom
2 |hello, Tom

```

I am done with the rollup part but how to filter the duplicate tokens?

Current code:

```
df.select("id","values")
  .groupBy("id")
  .agg(concat_ws(",", collect_list("values")))

```

**2nd part of the question:**
i also tried via SQL but its showing also the duplicates.

```
spark.sql("select id, concat_ws(' ' ,collect_set(values)) as values from data group by id ").show(false)
+---+----------------------------+
|id |values                      |
+---+----------------------------+
|1  |hello hello Sam hello Tom   |
|2  |hello hello Tom             |
+---+----------------------------+

```

How to get rid of duplicates in above query