title: How to process multiple parquet files in parallel using Pyspark?
body: <p>I am using Azure Databricks and i'm new to Pyspark and big data.</p>
  
  <p>Here is my problem:</p>
  
  <p>I have several parquet files in a directory on azure databricks.
  I want to read these files to a pyspark dataframe and use the drop duplicates method to remove duplicate rows - a QA check.
  I then want to overwrite these files in the same directory after dropping the duplicates.</p>
  
  <p>Currently, I am using a for loop to loop over each parquet file in the directory. However, this is an inefficient way of doing things. I was wondering if there is a way to process these parquet files in parallel to save computational time. If so how do I need to change my code.</p>
  
  <p>Here is the code:</p>
  
  <pre><code>for parquet_file_name in dir:
       df = spark.read.option("header", "true").option("inferschema", "false").parquet('{}/{}'.format(dir,parquet_file_name))
       df.dropDuplicates().write.mode('overwrite').parquet('{}/{}'.format(dir,parquet_file_name)
  </code></pre>
  
  <p>Any help here would be much appreciated.</p>
  
  <p>Many thanks.  </p>