Title: How to Calculate number of partitions for a spark RDD/DataSet
Tags: apache-spark;optimization;rdd;partitioning
Question: I want a formula of course Heuristic , but which can be used as a rule of thumb for deciding the following
Increase partitions to X partitions for optimal performance
and best utilisation of the cluster resources.
Decrease partitions to X partitions for optimal performance
and best utilisation of the cluster resources.
Don't repartition the data , default partition are good enough for optimal performance and best utilisation of cluster resources.
I have this information
No of nodes
No of cores
No of executors
Memory available per executor
Size of the Data
Total memory etc
What could be a rule of thumb , a formula to calculate the no of partitions are run time for best performance.
Say If I have rdd and its default partitions are "X" then after applying the formula using the above info I should be able to decide "y" by either increasing , decreasing or not repartitoning.
Of course there cannot be an exact formula for this , but definitely a close approach would also just be fine.
Thanks
Comment: I dont see any answer here. But recently I came across an excellent talk . At 30th minute he tries to give a formula which works in most of the cases. https://www.youtube.com/watch?v=_ArCesElWp8&t=4512s



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 1, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:10/27/17', 'FavoriteCount': 2.0, 'PostTypeId': 1, 'Tags': '<apache-spark><optimization><rdd><partitioning>', 'Title': 'How to Calculate number of partitions for a spark RDD/DataSet', 'ViewCount': 611, 'id': 46975509, 'user_AboutMe': '', 'user_AccountId': 5914315.0, 'user_DisplayName': 'Syed Ammar Mustafa', 'user_Reputation': 373.0, 'user_Views': 68.0, 'user_id': 4653573.0}}