In Spark, there are two ways of explicitly changing the number of partitions. We can use either the `repartition` function or `coalesce`. Both accept a numeric parameter, which tells Spark how many partitions we want to have. I will explain the difference between those two functions and when we should use them.

## Repartition

First of all, the `repartition` function that performs operations on Spark `Datasets` comes in three versions. There is a function that accepts only the number of partitions we want to get. In this case, Spark performs `RoundRobinPartitioning` to uniformly distribute the whole `Dataset` across the requested number of partitions.

If we specify both the expression and the number of partitions, Spark does `hashpartitioning`, which produces the requested number of partitions in which data is partitioned using the given expression.

Note that, the result will most likely not be uniformly distributed. We may end up with partitions that contain significantly more data if the partitioning value is skewed.

In addition to that, we may use the function that accepts only the partitioning expression and uses the `sql.shuffle.partitions` parameter as the number of partitions.

We see that the `repartition` function can both increase and decrease the number of partitions. Suppose we do not use the partitioning expression. In that case, we get partitions that distribute the workload equally across all worker nodes (unless we use another operation that implicitly repartitions the data using `hashpartitioning`).

## Coalesce

Coalesce is a little bit different. It accepts only one parameter - there is no way to use the partitioning expression, and it can only decrease the number of partitions. It works this way because we should use `coalesce` only to combine the existing partitions. It merges the data by draining existing partitions into others and removing the empty partitions.

Note that if we specify the number of partitions larger than the current partitioning, coalesce will not do anything.

## When to Use Coalesce Instead of Repartition?

According to the documentation, it is better to run `repartition` instead of `coalesce` when we want to do “drastic coalesce” (for example, merge everything into one partition). When we do it, the upstream calculations execute in parallel, so the entire computation is distributed across a larger number of workers.

I can think of only one use case when I am sure that the coalesce function is a better choice. If I have just used `where` or `filter`, and I think that some of the partitions may be “almost empty”, I will want to move the data around to have a similar number of rows in every partition. `coalesce` will not give me uniformed distribution, but it is going to be faster then `repartition`, and it should be good enough in this case.

## Is Coalesce Worth Consideration In Practice?

Most likely no. Usually, the Spark jobs are full of the `group by` and `join` operations, so avoiding full reshuffles using coalesce in a few places will not make a huge difference.

Older post

## How to pivot an Apache Spark DataFrame

IMPORTANT FOR SEO!!!
{"ft_lang_label":"__label__en","ft_lang_prob":0.89024013,"math_prob":0.87673455,"size":3006,"snap":"2023-40-2023-50","text_gpt3_token_len":609,"char_repetition_ratio":0.19320454,"word_repetition_ratio":0.012295082,"special_character_ratio":0.18330006,"punctuation_ratio":0.09375,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.95341593,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-09-25T05:42:26Z\",\"WARC-Record-ID\":\"<urn:uuid:09b69717-c143-4261-a121-229035020374>\",\"Content-Length\":\"82763\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:aaf33002-2a86-45f5-9db1-c133420a86f1>\",\"WARC-Concurrent-To\":\"<urn:uuid:e03ff294-3a10-4801-9117-5f27d9904fee>\",\"WARC-IP-Address\":\"18.160.18.25\",\"WARC-Target-URI\":\"https://www.mikulskibartosz.name/what-is-the-difference-between-repartition-and-coalesce-in-apache-spark\",\"WARC-Payload-Digest\":\"sha1:3CIMHFQAWNAH2XXJQ6YEQVNZ7I46GN4C\",\"WARC-Block-Digest\":\"sha1:IS2OEIVW4HK4HSQQCZNE7J43HX5DMLI2\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-40/CC-MAIN-2023-40_segments_1695233506686.80_warc_CC-MAIN-20230925051501-20230925081501-00606.warc.gz\"}"}