Save only the required CSV file using PySpark
Those are default Log files created when saving from PySpark . We can't eliminate this.
Using coalesce(1) you can save in a single file without partition.
I am quite new to PySpark, I am trying to read and then save a CSV file using Azure Databricks.
After saving the file I see many other files like "_Committed","_Started","_Success" and finally the CSV file with a totally different name.
I have already checked using DataFrame repartition(1) and coalesce(1) but this only deals when the CSV file itself was partitioned by Spark. Is there anything that can be done using PySpark?