title: Batch Map about 3000 Parquet files in Hadoop separately, how do it efficient?
body: <h2>About the data</h2>
  
  <p>The data used for analysis is saved as parquet file on Hadoop, about 3000+ </p>
  
  <h2>About the task</h2>
  
  <p>I want batch map those files use a map function, each file is separated, the map function use pandas dataframe to get the result, then I need collect all the map result.</p>
  
  <h2>My poor code...</h2>
  
  <pre><code>def map_func(df):
  ...:     pdf = df.toPandas()
  ...:     return some_other_func(pdf)
  
  #first load data
  df1 = spark.read.load("/data/file1")
  result1 = df1.map(map_func)
  
  ...
  
  dfN = spark.read.load("/data/fileN")
  resultN = dfN.map(map_func)
  
  final = (result1, ... , resultN)
  
  # How to batch the above process 
  </code></pre>
  
  <p>How can I simply this task efficiently?</p>