Is it good idea to repartition 50 million records data in dataframe? If yes then someone please tell me the appropriate way of doing this
Depending on your machine try maintaining a fixed number of partitions. It is always a good idea to partition but in most cases, it's not a good idea to partition based on date(Not sure because I don't know the nature of your data).
We are going to handle Big Data (~50 million records) in our organization. We are partitioning data on the basis of date and other some parameters, but data is not equally partitioned. Can we do repartition on it for good performance?