title: Convert CSV files from multiple directory into parquet in PySpark
body: <p>I have CSV files from multiple paths that are not parent directories in s3 bucket. All the tables have the same partition keys. </p>
  
  <p>the directory of the s3:</p>
  
  <pre><code>table_name_1/partition_key_1 = &lt;pk_1&gt;/partition_key_2 = &lt;pk_2&gt;/file.csv
  table_name_2/partition_key_1 = &lt;pk_1&gt;/partition_key_2 = &lt;pk_2&gt;/file.csv
  ...
  </code></pre>
  
  <p>I need to convert these csv files into parquet files and store them in another s3 bucket that has the same directory structure. </p>
  
  <p>the directory of another s3:</p>
  
  <pre><code>table_name_1/partition_key_1 = &lt;pk_1&gt;/partition_key_2 = &lt;pk_2&gt;/file.parquet
  table_name_2/partition_key_1 = &lt;pk_1&gt;/partition_key_2 = &lt;pk_2&gt;/file.parquet
  ...
  </code></pre>
  
  <p>I have a solution is iterating through the s3 bucket and find the CSV file and convert it to parquet and save to the another S3 path. I find this way is not efficient, because i have a loop and did the conversion one file by one file. </p>
  
  <p>I want to utilize the spark library to improve the efficiency. 
  Then, I tried: </p>
  
  <pre><code>spark.read.csv('s3n://bucket_name/table_name_1/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/table_name_1')
  </code></pre>
  
  <p>This way works good for each table, but to optimize it more, I want to take the table_name as a parameter, something like: </p>
  
  <pre><code>TABLE_NAMES = [table_name_1, table_name_2, ...]
  spark.read.csv('s3n://bucket_name/{*TABLE_NAMES}/').write.partitionBy('partition_key_1', 'partition_key_2').parquet('s3n://another_bucket/{*TABLE_NAMES}')
  </code></pre>
  
  <p>Thanks</p>