# PySpark: How to Perform Union and Return Distinct Rows

You can use the following syntax to perform a union on two PySpark DataFrames and return only distinct rows:

```df_union_distinct = df1.union(df2).distinct()
```

This particular example performs a union between the PySpark DataFrames named df1 and df2 and returns only the distinct rows between the two DataFrames.

The following example shows how to use this syntax in practice.

## Example: How to Perform Union and Return Distinct Rows in PySpark

Suppose we have the following PySpark DataFrame named df1:

```from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

#define data
data1 = [['A', 'East', 11],
['B', 'East', 8],
['C', 'East', 31],
['D', 'West', 16],
['E', 'West', 6]]

#define column names
columns1 = ['team', 'conference', 'points']

#create DataFrame
df1 = spark.createDataFrame(data1, columns1)

#view DataFrame
df1.show()

+----+----------+------+
|team|conference|points|
+----+----------+------+
|   A|      East|    11|
|   B|      East|     8|
|   C|      East|    31|
|   D|      West|    16|
|   E|      West|     6|
+----+----------+------+
```

And suppose we have another DataFrame named df2:

```#define data
data2 = [['A', 'East', 11],
['B', 'East', 8],
['G', 'East', 31],
['H', 'West', 16]]

#define column names
columns2 = ['team', 'conference', 'points']

#create DataFrame
df2 = spark.createDataFrame(data2, columns2)

#view DataFrame
df2.show()

+----+----------+------+
|team|conference|points|
+----+----------+------+
|   A|      East|    11|
|   B|      East|     8|
|   G|      East|    31|
|   H|      West|    16|
+----+----------+------+
```

Suppose we use the following syntax to perform a union between these two DataFrames:

```#perform union with df1 and df2
df_union = df1.union(df2)

#view final DataFrame
df_union.show()

+----+----------+------+
|team|conference|points|
+----+----------+------+
|   A|      East|    11|
|   B|      East|     8|
|   C|      East|    31|
|   D|      West|    16|
|   E|      West|     6|
|   A|      East|    11|
|   B|      East|     8|
|   G|      East|    31|
|   H|      West|    16|
+----+----------+------+```

Notice that the rows with team A and B from each DataFrame are shown in the final DataFrame, even though these rows are duplicates.

To perform a union and only return the distinct rows, we can use the following syntax:

```#perform union with df1 and df2 and return only distinct rows
df_union_distinct = df1.union(df2).distinct()

#view final DataFrame
df_union_distinct.show()

+----+----------+------+
|team|conference|points|
+----+----------+------+
|   A|      East|    11|
|   B|      East|     8|
|   C|      East|    31|
|   D|      West|    16|
|   E|      West|     6|
|   G|      East|    31|
|   H|      West|    16|
+----+----------+------+
```

Notice that we have successfully performed a union between the two DataFrames and only the distinct rows are returned.

Note: You can find the complete documentation for the PySpark union function here.
{"ft_lang_label":"__label__en","ft_lang_prob":0.6866453,"math_prob":0.78391397,"size":2919,"snap":"2023-40-2023-50","text_gpt3_token_len":821,"char_repetition_ratio":0.20994854,"word_repetition_ratio":0.27803737,"special_character_ratio":0.3788969,"punctuation_ratio":0.12711865,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9712078,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-12-01T12:42:50Z\",\"WARC-Record-ID\":\"<urn:uuid:e120cf6a-2cd9-4846-8119-6e33a4e99232>\",\"Content-Length\":\"47195\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:c4384f07-1384-4ddd-b94d-65c2bcbe4b53>\",\"WARC-Concurrent-To\":\"<urn:uuid:d2dba34f-2860-4fac-a9f1-459d042d7d66>\",\"WARC-IP-Address\":\"35.190.31.54\",\"WARC-Target-URI\":\"https://www.statology.org/pyspark-union-distinct/\",\"WARC-Payload-Digest\":\"sha1:7UPJ4EEVWJN4POZEQISVNBDWDNKKE4FB\",\"WARC-Block-Digest\":\"sha1:BX2DW4NXT5TXIZNOWTYB637WECWCRD44\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-50/CC-MAIN-2023-50_segments_1700679100287.49_warc_CC-MAIN-20231201120231-20231201150231-00772.warc.gz\"}"}