title: Spark step on EMR just hangs as "Running" after done writing to S3
body: <p>Running PySpark 2 job on EMR 5.1.0 as a step. Even after the script is done with a <code>_SUCCESS</code> file written to S3 and Spark UI showing the job as completed, EMR still shows the step as "Running". I've waited for over an hour to see if Spark was just trying to clean itself up but the step never shows as "Completed". The last thing written in the logs is:</p>
  
  <pre><code>INFO MultipartUploadOutputStream: close closed:false s3://mybucket/some/path/_SUCCESS
  INFO DefaultWriterContainer: Job job_201611181653_0000 committed.
  INFO ContextCleaner: Cleaned accumulator 0
  </code></pre>
  
  <p>I didn't have this problem with Spark 1.6. I've tried a bunch of different <code>hadoop-aws</code> and <code>aws-java-sdk</code> jars to no avail.</p>
  
  <p>I'm using the default Spark 2.0 configurations so I don't think anything else like metadata is being written. Also the size of the data doesn't seem to have an impact on this problem.</p>