What is the logic behind "compute()" in Dask dataframes?
Each partition in a Dask DataFrame is a Pandas DataFrame.
compute() combines all the partitions (Pandas DataFrames) into a single Pandas DataFrame.
Dask is fast because it can perform computations on partitions in parallel.  Pandas can be slower because it only works on one partition.
You should avoid calling compute() whenever possible.  It better to have the data spread out in multiple partitions, so it can be processed in parallel.
In rare cases you can compute to Pandas (e.g. when doing a large to small join or after a huge filtering operation), but it's best to learn how to use the Dask API to run computations in parallel.
I'm struggling to understand when and when not to use compute() in Dask dataframes. I usually write my code by adding/removing compute() until the code works, but that's extremely error-prone. How should I use compute() in Dask? Does it differ in Dast Distributed?