title: pyspark on emr with boto3, copy of s3 object result with Futures timed out after [100000 milliseconds]
body: <p>I have a pyspark application that will transform csv to parquet and before this happen I'm copying some S3 object from a bucket to another.</p>
  
  <p>pyspark with spark 2.4, emr 5.27, maximizeResourceAllocation set to true</p>
  
  <p>I have various csv files size, from 80kb to 500mb.</p>
  
  <p>Nonetheless, my EMR cluster (it doesn't fail on local with spark-submit) fails at 70% completion on a file that is 166mb (a previous at 480mb succeeded).</p>
  
  <p>The job is simple:</p>
  
  <pre><code>def organise_adwords_csv():
      s3 = boto3.resource('s3')
      bucket = s3.Bucket(S3_ORIGIN_RAW_BUCKET)
      for obj in bucket.objects.filter(Prefix=S3_ORIGIN_ADWORDS_RAW + "/"):
          key = obj.key
  
          copy_source = {
              'Bucket': S3_ORIGIN_RAW_BUCKET,
              'Key': key
          }
  
          key_tab = obj.key.split("/")
          if len(key_tab) &lt; 5:
              print("continuing from length", obj)
              continue
          file_name = ''.join(key_tab[len(key_tab)-1:len(key_tab)])
          if file_name == '':
              print("continuing", obj)
              continue
  
          table = file_name.split("_")[1].replace("-", "_")
          new_path = "{0}/{1}/{2}".format(S3_DESTINATION_ORDERED_ADWORDS_RAW_PATH, table, file_name)
          print("new_path", new_path) &lt;- the last print will end here
          try:
              s3.meta.client.copy(copy_source, S3_DESTINATION_RAW_BUCKET, new_path)
              print("copy done")
          except Exception as e:
              print(e)
              print("an exception occured while copying")
  
  if __name__=='__main__':
      organise_adwords_csv()
  
      print("copy Final done") &lt;- never printed
  
      spark = SparkSession.builder.appName("adwords_transform") \
      ...
  </code></pre>
  
  <p>but, in the stdout, no  errors / exception are showing.</p>
  
  <p>In stderr logs:</p>
  
  <pre><code>19/10/09 16:16:57 INFO ApplicationMaster: Waiting for spark context initialization...
  19/10/09 16:18:37 ERROR ApplicationMaster: Uncaught exception: 
  java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
      at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
      at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
      at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
      at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:468)
      at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:779)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
      at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:778)
      at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
      at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:803)
      at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
  19/10/09 16:18:37 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
      at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
      at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
      at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:220)
      at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:468)
      at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:779)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
      at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:778)
      at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
      at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:803)
      at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
  )
  19/10/09 16:18:37 INFO ShutdownHookManager: Shutdown hook called
  </code></pre>
  
  <p>I'm completely blind, I don't understand what is failing / why.
  How can I figure that out? On local it works like a charm (but super slow of course)</p>
  
  <p><strong>Edit:</strong>
  After many tries I can confirm that the function:</p>
  
  <pre><code>s3.meta.client.copy(copy_source, S3_DESTINATION_RAW_BUCKET, new_path)
  </code></pre>
  
  <p>make the EMR cluster timeout, even tho it processed 80% of the files already.
  Does anyone have a recommendation about this?</p>