title: Using AWS Glue to convert very big csv.gz (30-40 gb each) to parquet
body: <p>There are lots of such questions but nothing seems to help. I am trying to covert quite large csv.gz files to parquet and keep on getting various errors like</p>
  
  <pre><code>'Command failed with exit code 1'
  </code></pre>
  
  <p>or</p>
  
  <pre><code>An error occurred while calling o392.pyWriteDynamicFrame. Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-172-31-5-241.eu-central-1.compute.internal, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container marked as failed
  </code></pre>
  
  <p>. In the metrics monitoring I don't see much of CPU or Memory load. There is ETL data movement but that should be trigger any error when working with S3. </p>
  
  <p>Another problem is that such job runs 4-5 hours before throwing. Is it an expected behavior? CSV files have like 30-40 cols.</p>
  
  <p>I don't know which direction to go. Can Glue overall handle such large files?</p>