title: aws Glue job: how to merge multiple output .csv files in s3
body: <p>I created an aws Glue Crawler and job. The purpose is to transfer data from a postgres RDS database table to one single .csv file in S3. Everything is working, but I get a total of 19 files in S3. Every file is empty except three with one row of the database table in it as well as the headers. So every row of the database is written to a seperate .csv file.
  What can I do here to specify that I want only one file where the first row are the headers and afterwards every line of the database follows?</p>
  
  <pre><code>import sys
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkContext
  from awsglue.context import GlueContext
  from awsglue.job import Job
  
  ## @params: [JOB_NAME]
  args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  
  sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args['JOB_NAME'], args)
  ## @type: DataSource
  ## @args: [database = "gluedatabse", table_name = "postgresgluetest_public_account", transformation_ctx = "datasource0"]
  ## @return: datasource0
  ## @inputs: []
  datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "gluedatabse", table_name = "postgresgluetest_public_account", transformation_ctx = "datasource0")
  ## @type: ApplyMapping
  ## @args: [mapping = [("password", "string", "password", "string"), ("user_id", "string", "user_id", "string"), ("username", "string", "username", "string")], transformation_ctx = "applymapping1"]
  ## @return: applymapping1
  ## @inputs: [frame = datasource0]
  applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("user_id", "string", "user_id", "string"), ("username", "string", "username", "string"),("password", "string", "password", "string")], transformation_ctx = "applymapping1")
  ## @type: DataSink
  ## @args: [connection_type = "s3", connection_options = {"path": "s3://BUCKETNAMENOTSHOWN"}, format = "csv", transformation_ctx = "datasink2"]
  ## @return: datasink2
  ## @inputs: [frame = applymapping1]
  datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://BUCKETNAMENOTSHOWN"}, format = "csv", transformation_ctx = "datasink2")
  job.commit()
  </code></pre>
  
  <p>The database looks like this:
  <a href="https://i.stack.imgur.com/p1dvW.png" rel="nofollow noreferrer">Databse picture</a></p>
  
  <p>It looks like this in S3:
  <a href="https://i.stack.imgur.com/oUoI6.png" rel="nofollow noreferrer">S3 Bucket</a></p>
  
  <p>One sample .csv in the S3 looksl ike this:</p>
  
  <pre><code>password,user_id,username
  346sdfghj45g,user3,dieter
  </code></pre>
  
  <p>As I said, there is one file for each table row.</p>
  
  <p>Edit:
  The multipartupload to s3 doesn't seem to work correctly. It just uplaods the parts but doesn't merge them together when finished. Here are the last lines of the job log:
  Here are the last lines of the log:</p>
  
  <pre><code>19/04/04 13:26:41 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
  19/04/04 13:26:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
  19/04/04 13:26:41 INFO Executor: Finished task 16.0 in stage 2.0 (TID 18). 2346 bytes result sent to driver
  19/04/04 13:26:41 INFO MultipartUploadOutputStream: close closed:false s3://bucketname/run-1554384396528-part-r-00018
  19/04/04 13:26:41 INFO MultipartUploadOutputStream: close closed:true s3://bucketname/run-1554384396528-part-r-00017
  19/04/04 13:26:41 INFO MultipartUploadOutputStream: close closed:false s3://bucketname/run-1554384396528-part-r-00019
  19/04/04 13:26:41 INFO Executor: Finished task 17.0 in stage 2.0 (TID 19). 2346 bytes result sent to driver
  19/04/04 13:26:41 INFO MultipartUploadOutputStream: close closed:true s3://bucketname/run-1554384396528-part-r-00018
  19/04/04 13:26:41 INFO Executor: Finished task 18.0 in stage 2.0 (TID 20). 2346 bytes result sent to driver
  19/04/04 13:26:41 INFO MultipartUploadOutputStream: close closed:true s3://bucketname/run-1554384396528-part-r-00019
  19/04/04 13:26:41 INFO Executor: Finished task 19.0 in stage 2.0 (TID 21). 2346 bytes result sent to driver
  19/04/04 13:26:41 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
  19/04/04 13:26:41 INFO CoarseGrainedExecutorBackend: Driver from 172.31.20.76:39779 disconnected during shutdown
  19/04/04 13:26:41 INFO CoarseGrainedExecutorBackend: Driver from 172.31.20.76:39779 disconnected during shutdown
  19/04/04 13:26:41 INFO MemoryStore: MemoryStore cleared
  19/04/04 13:26:41 INFO BlockManager: BlockManager stopped
  19/04/04 13:26:41 INFO ShutdownHookManager: Shutdown hook called
  End of LogType:stderr
  </code></pre>