Overwrite in to same partition files after transformation based on the filename using spark
Well, I'm not sure if my answer is the best, but I hope it is.
Basically To write output to file, Spark repects hadoop config, which is mapreduce.output.basename
Default value should be something like part-00000.
You can adjust this config but can't make exactly same as your file name convention.
So you have to write and rename to your file name convention.
So procedure is simple.

Write file to the path.
Rename output file to original name(may delete old and rename)
Hi I have files in a S3 bucket
MyBucket/object/file 1.csv, file 2.csv, file 3.csv,
I have loaded this data into single dataframe and need to do some transformation based on columns.Then I want to write to transform column values now I want to overwrite the files back in to same file1.csv, file2.csv,file3.csv.
When I give overwrite commands its creating another file in same folder and loading values
How to write function or code using python and spark or scala