title: Read multiple groups of csv files from a folder and insert to respective target tables parallelly using spark or databricks
body: <p><strong>Input</strong>:
  abc.tar.gz -&gt; un tar - &gt; Folder: abc</p>
  <p><strong>Folder structure of abc:</strong></p>
  <p>root folder: abc contains csv files that are generated from 100 cities every 5 minutes in day.</p>
  <p>Number of csv files: 100 cities * 12 files per hour * 24 hours = 28800 csv files</p>
  <pre><code>abc/    
  city1_0005.csv
  city1_0010.csv
  ..
  city1_2355.csv
  ..
  ..
  city2_0005.csv
  city2_0010.csv
  ..
  city2_2355.csv
  ..
  ..
  city100_0005.csv
  city100_0010.csv
  </code></pre>
  <p><strong>Functional Requirement:</strong></p>
  <ul>
  <li>Using spark/ databricks Create table for each city and load the respective city csv files (288) into table. total 100 tables will be there in target location. 1 table for 1 city.</li>
  <li>Each city has different schema .All columns are different for each city. so I cannot write all cities data into single table with partitions.</li>
  </ul>
  <p><strong>technical requirement:</strong>
  Read and process the files parallelly for better performance</p>
  <hr />
  <p>I have developed below code to process data sequentially.
  I am looking for ways to optimize it .</p>
  <pre><code>staging_path=&quot;abfss://xyz/abc&quot;
  
  #using databricks utils  to get the list of files in folder
  filesProp = dbutils.fs.ls(staging_adls_path)
  
  #extracting the city names from list of filenames
  filesSet  =set()
  for file in filesProp:
      filesSet.add(file.name.split('-')[0])
  
  #empty list to store dataframes
  dictionary_df = {} 
  
  #reading 1 city data and inserting to table
  for fileName in filesSet:
      filePath = staging_path+fileName+&quot;*&quot;
      print(filePath)
      dictionary_df[fileName] = spark.read.options(header='True', delimiter=',').csv(filePath)
      dictionary_df[fileName].write.saveAsTable(fileName) 
  </code></pre>