title: setting up s3 for logs in airflow
body: <p>I am using docker-compose to set up a scalable airflow cluster. I based my approach off of this Dockerfile <a href="https://hub.docker.com/r/puckel/docker-airflow/" rel="noreferrer">https://hub.docker.com/r/puckel/docker-airflow/</a></p>
  
  <p>My problem is getting the logs set up to write/read from s3. When a dag has completed I get an error like this</p>
  
  <pre><code>*** Log file isn't local.
  *** Fetching here: http://ea43d4d49f35:8793/log/xxxxxxx/2017-06-26T11:00:00
  *** Failed to fetch log file from worker.
  
  *** Reading remote logs...
  Could not read logs from s3://buckets/xxxxxxx/airflow/logs/xxxxxxx/2017-06-
  26T11:00:00
  </code></pre>
  
  <p>I set up a new section in the <code>airflow.cfg</code> file like this</p>
  
  <pre><code>[MyS3Conn]
  aws_access_key_id = xxxxxxx
  aws_secret_access_key = xxxxxxx
  aws_default_region = xxxxxxx
  </code></pre>
  
  <p>And then specified the s3 path in the remote logs section in <code>airflow.cfg</code></p>
  
  <pre><code>remote_base_log_folder = s3://buckets/xxxx/airflow/logs
  remote_log_conn_id = MyS3Conn
  </code></pre>
  
  <p>Did I set this up properly and there is a bug? Is there a recipe for success here that I am missing?</p>
  
  <p>-- Update</p>
  
  <p>I tried exporting in URI and JSON formats and neither seemed to work. I then exported the aws_access_key_id and aws_secret_access_key and then airflow started picking it up. Now I get his error in the worker logs </p>
  
  <pre><code>6/30/2017 6:05:59 PMINFO:root:Using connection to: s3
  6/30/2017 6:06:00 PMERROR:root:Could not read logs from s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
  6/30/2017 6:06:00 PMERROR:root:Could not write logs to s3://buckets/xxxxxx/airflow/logs/xxxxx/2017-06-30T23:45:00
  6/30/2017 6:06:00 PMLogging into: /usr/local/airflow/logs/xxxxx/2017-06-30T23:45:00
  </code></pre>
  
  <p>-- Update</p>
  
  <p>I found this link as well 
  <a href="https://www.mail-archive.com/dev@airflow.incubator.apache.org/msg00462.html" rel="noreferrer">https://www.mail-archive.com/dev@airflow.incubator.apache.org/msg00462.html</a></p>
  
  <p>I then shelled into one of my worker machines (separate from the webserver and scheduler) and ran this bit of code in python</p>
  
  <pre><code>import airflow
  s3 = airflow.hooks.S3Hook('s3_conn')
  s3.load_string('test', airflow.conf.get('core', 'remote_base_log_folder'))
  </code></pre>
  
  <p>I receive this error.</p>
  
  <pre><code>boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
  </code></pre>
  
  <p>I tried exporting several different types of <code>AIRFLOW_CONN_</code> envs as explained here in the connections section <a href="https://airflow.incubator.apache.org/concepts.html" rel="noreferrer">https://airflow.incubator.apache.org/concepts.html</a> and by other answers to this question.</p>
  
  <pre><code>s3://&lt;AWS_ACCESS_KEY_ID&gt;:&lt;AWS_SECRET_ACCESS_KEY&gt;@S3
  
  {"aws_account_id":"&lt;xxxxx&gt;","role_arn":"arn:aws:iam::&lt;xxxx&gt;:role/&lt;xxxxx&gt;"}
  
  {"aws_access_key_id":"&lt;xxxxx&gt;","aws_secret_access_key":"&lt;xxxxx&gt;"}
  </code></pre>
  
  <p>I have also exported AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with no success.</p>
  
  <p>These credentials are being stored in a database so once I add them in the UI they should be picked up by the workers but they are not able to write/read logs for some reason.</p>