title: Converting a S3 data folder into bucketed and partitioned data store
body: <p>I have a folder with TBs of structured data and they all adher to a fixed schema. The folder structure is as such:</p>
  
  <pre><code>s3://main-bucket/
                  folder-1/ &lt;= One folder will only contain data from one algorithm
                                  part-0000-1.csv &lt;= Created on 06/01/2019 by algorithm1
                                  part-0000-2.csv &lt;= Created on 06/01/2019 by algorithm1
                  folder-2/
                                  part-1000-1.csv &lt;= Created on 06/02/2019 by algorithm2
                                  part-1000-2.csv &lt;= Created on 06/02/2019 by algorithm2
  </code></pre>
  
  <p>Querying on main-bucket using Spark or Athena is very inefficient and I wanted to convert it a format like this:</p>
  
  <pre><code>s3://data-bucket/
                  algorithm1/ &lt;= derived from a column in folder-1/part files and all rows in this folder has the same algorithm value
                    2019/                   
                      06/
                        01/
                          part-0001.parquet
                  algorithm2/
                      2019/                   
                         06/
                           02/
                              part-0001.parquet
  </code></pre>
  
  <p>Anyway to do this?</p>