I have a dataframe which has 500 partitions and is shuffled.
I want to repartition it based on one column say 'city'
But the city column is extremely skewed as it has only three possible values.
So when I repartition​ based on column city, even if I specify 500 number of partitions, only three are getting data. Because of this I am running into performance issues.
I searched on internet but could not find any suitable solution.
Is there a way to repartition the dataframe uniformly across partitions based in city column.
What I need is: city1 goes to say first 5 partitions, city2 goes to next 490 partitions and city3 goes to remaining 5 partitions.