Title: How to move files written with Pandas on Spark cluster to HDFS?
Tags: apache-spark;hadoop;pyspark
Question: I'm running a Spark Job using Cluster Mode and writing few files using Pandas and I think it's writing in temp directory, now I want to move these files or write these files in HDFS.

Here is the accepted answer: You have multiple options:
convert Pandas Dataframe into PySpark DataFrame and simply save it into HDFS
spark_df = spark.createDataFrame(pandas_df)
spark_df.write.parquet(&quot;hdfs:///path/on/hdfs/file.parquet&quot;)
save file locally using Pandas and use subprocess to copy file to HDFS
import subprocess
command = &quot;hdfs dfs -copyFromLocal -f local/file.parquet /path/on/hdfs&quot;.split()
result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(result.stdout)
print(result.stderr)
save file locally and use 3rd party library - hdfs3 - to copy file to HDFS
from hdfs3 import HDFileSystem
hdfs = HDFileSystem()
hdfs.cp(&quot;local/file.parquet&quot;, &quot;/path/on/hdfs&quot;)


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2021:00:02/16/21', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<apache-spark><hadoop><pyspark>', 'Title': 'How to move files written with Pandas on Spark cluster to HDFS?', 'ViewCount': 298, 'id': 66229873, 'user_AboutMe': "<p>Hi, I'm a Cloud Data Engineer with more than 1 year of experience in Amazon Web Services like Glue, EMR, Lambda, Athena, S3, EC2, RDS, DynamoDB, Quick Sight, IAM, VPC, Kinesis etc. I have experience 1+ year of experience in Spark including Structured Streaming, RDDs, Dataframes, Spark SQL.</p>\n", 'user_AccountId': 17183124.0, 'user_DisplayName': 'Abdul Haseeb', 'user_Reputation': 404.0, 'user_Views': 125.0, 'user_id': 12438249.0}}