Is it good idea to repartition 50 million records data in dataframe? If yes then someone please tell me the appropriate way of doing this
Usually, partitioning is a good idea and as @Karthik already said, often the date is not the best idea. In my experience it always made sense to partition your data based on the amount of workers you have. So ideally your partition size is a multiple of your workers. We normally use 120 partitions, as we have 24 workers in our spark environment and end up with code like:
new_df = spark.read.csv("some_csv.csv", header="true", escape="\"", quote="\"").repartition(100)
We also experienced way better performance in working with parquet instead of csv, which is a tradeoff, as the data has to be read, repartitioned and stored again, but it paid off in the analysis steps. So maybe you should also consider this conversion.
We are going to handle Big Data (~50 million records) in our organization. We are partitioning data on the basis of date and other some parameters, but data is not equally partitioned. Can we do repartition on it for good performance?