title: AWS download all Spark "part-*" files and merge them into a single local file
body: <p>I've run a Spark job via databricks on AWS, and by calling</p>
  
  <pre><code>big_old_rdd.saveAsTextFile("path/to/my_file.json")
  </code></pre>
  
  <p>have saved the results of my job into an S3 bucket on AWS. The result of that spark command is a directory <code>path/to/my_file.json</code> containing portions of the result:</p>
  
  <pre><code>_SUCCESS
  part-00000
  part-00001
  part-00002
  </code></pre>
  
  <p>and so on. I can copy those part files to my local machine using the <a href="https://aws.amazon.com/cli/" rel="nofollow">AWS CLI</a> with a relatively simple command:</p>
  
  <pre><code>aws s3 cp s3://my_bucket/path/to/my_file.json local_dir --recursive
  </code></pre>
  
  <p>and now I've got all those <code>part-*</code> files locally. Then I can get a single file with </p>
  
  <pre><code>cat $(ls part-*) &gt; result.json
  </code></pre>
  
  <p>The problem is that this two-stage process is cumbersome and leaves file parts all over the place. I'd like to find a single command that will download and merge the files (ideally in order). When dealing with HDFS directly this is something like <code>hadoop fs -cat "path/to/my_file.json/*" &gt; result.json</code>.</p>
  
  <p>I've looked around through the AWS CLI documentation but haven't found an option to merge the file parts automatically, or to cat the files. I'd be interested in either some fancy tool in the AWS API or some bash magic that will combine the above commands.</p>
  
  <p>Note: Saving the result into a single file via spark is not a viable option as this requires coalescing the data to a single partition during the job. Having multiple part files on AWS is fine, if not desirable. But when I download a local copy, I'd like to merge.</p>