title: How to save and use Spark History Server logs in AWS S3
body: <p>I want to record and view Event Log of Spark History Server in AWS S3.</p>
  <p>The following are the properties recorded in spark-defaults.conf.</p>
  <pre><code>spark.hadoop.fs.s3a.impl          org.apache.hadoop.fs.s3a.S3AFileSystem
  spark.hadoop.fs.s3a.endpoint    {{endpoint}}
  spark.hadoop.fs.s3a.access.key  {{accessKey}}
  spark.hadoop.fs.s3a.secret.key  {{secretKey}}
  spark.hadoop.fs.s3a.fast.upload true
  spark.hadoop.fs.s3a.block.size  268435456
  spark.eventLog.enabled            true
  spark.eventLog.dir                s3a://{{bucketName}}/{{path}}
  spark.history.fs.logDirectory     s3a://{{bucketName}}/{{path}}
  </code></pre>
  <p>However, when starting the spark history server, the following error occurs.</p>
  <pre><code>20/10/07 14:07:14 INFO S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden
  20/10/07 14:07:14 INFO S3AFileSystem: HTTP Status Code: 403
  20/10/07 14:07:14 INFO S3AFileSystem: AWS Error Code: null
  20/10/07 14:07:14 INFO S3AFileSystem: Error Type: Client
  20/10/07 14:07:14 INFO S3AFileSystem: Request ID: {{requestId}}
  20/10/07 14:07:14 INFO S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception
  Exception in thread &quot;main&quot; java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:296)
          at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
  Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: {{requestId}}
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
          at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
          at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
          at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
          at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:688)
          at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:71)
          at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:257)
          at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:211)
          at org.apache.spark.deploy.history.FsHistoryProvider.&lt;init&gt;(FsHistoryProvider.scala:207)
          at org.apache.spark.deploy.history.FsHistoryProvider.&lt;init&gt;(FsHistoryProvider.scala:86)
          ... 6 more
  </code></pre>
  <p>Uploading and downloading are performed well when calling APIs using the above authentication information.</p>
  <p>Did I miss anything when configuring Spark History Server?</p>