Show distinct column values in pyspark dataframe
You can use df.dropDuplicates(['col1','col2']) to get only distinct rows based on colX in the array.
With pyspark dataframe, how do you do the equivalent of Pandas df['col'].unique().
I want to list out all the unique values in a pyspark dataframe column.
Not the SQL type way (registertemplate then SQL query for distinct values).
Also I don't need groupby then countDistinct, instead I want to check distinct VALUES in that column.