# Distinct value of a column in pyspark – distinct()

In order to get the distinct value of a column in pyspark we will be using select() and distinct() function. There is another way to get distinct value of the column in pyspark using dropDuplicates() function. Distinct value of multiple columns in pyspark using dropDuplicates() function. Distinct value or unique value all the columns.  Let’s see with an example for both

• Distinct value of a column in pyspark using distinct() function
• Distinct value of the column in pyspark using dropDuplicates() function
• Unique/Distinct value of multiple columns in pyspark distinct() function & dropDuplicates() function
• unique/Distinct value of all the columns using distinct() function We will be using dataframe Basket_df #### Get distinct value of a column in pyspark – distinct() – Method 1

Distinct value of the column is obtained by using select() function along with distinct() function. select() function takes up the column name as argument, Followed by distinct() function will give distinct value of the column

```### Get distinct value of column

```

distinct value of “Price” column will be #### Get distinct value of a column – dropDuplicates() – Method 2

dropDuplicates() function takes up the column name as argument, will give distinct value of that column.

```### Drop Duplicate of the column
from pyspark.sql import Row
```

distinct value of “Price” column will be #### Distinct Value of multiple columns in pyspark: Method 1

Distinct value of the column in pyspark is obtained by using select() function along with distinct() function. select() function takes up mutiple column names as argument, Followed by distinct() function will give distinct value of those columns combined.

```### Get distinct value of multiple columns

```

distinct value of “Item_group” & “Price” columns will be #### Get distinct value of multiple columns in pyspark – Method 2

dropDuplicates() function takes up multiple column names as argument, will give distinct value of those columns.

```### Drop Duplicate of the column
from pyspark.sql import Row

```

distinct value of “Item_group” & “Price” columns will be #### distinct value of all the columns in pyspark using – distinct() function : Method 1

distinct() function without any arguments or select function, will give distinct value of the dataframe i.e. distinct value of the columns

```### distinct of all the columns

```

distinct value of all the columns will be #### distinct value of all the columns using dropDuplicates() function : Method 2

dropDuplicates() function without any arguments gets the distinct value of  all the columns as shown below.

```### Drop Duplicates of all the columns
from pyspark.sql import Row
{"ft_lang_label":"__label__en","ft_lang_prob":0.57250565,"math_prob":0.9782319,"size":3914,"snap":"2020-45-2020-50","text_gpt3_token_len":857,"char_repetition_ratio":0.27851662,"word_repetition_ratio":0.28524047,"special_character_ratio":0.21972407,"punctuation_ratio":0.07198748,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9757082,"pos_list":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"im_url_duplicate_count":[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-11-29T10:28:09Z\",\"WARC-Record-ID\":\"<urn:uuid:810fee85-218c-4358-99ba-9e4517544649>\",\"Content-Length\":\"70998\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:b55c5ef0-6699-43c0-b8f2-3a35e5d4d5e5>\",\"WARC-Concurrent-To\":\"<urn:uuid:8b2710ec-58ed-4c2b-b80f-6d17673d41a5>\",\"WARC-IP-Address\":\"172.67.190.10\",\"WARC-Target-URI\":\"https://www.datasciencemadesimple.com/distinct-value-of-a-column-in-pyspark/\",\"WARC-Payload-Digest\":\"sha1:6UMFV77TJTCRDPIYDBBAKCG3RR264UCJ\",\"WARC-Block-Digest\":\"sha1:JUT2IL3KGYBDD7OHZYQZFMUQFTEQTP6G\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-50/CC-MAIN-2020-50_segments_1606141197593.33_warc_CC-MAIN-20201129093434-20201129123434-00627.warc.gz\"}"}