title: boto3 s3 file upload using IAM role for authentication
body: <p>I made a code to upload the files to S3 using boto3. The code runs in docker using cron job.
  Initially I've set the aws credentials in the Dockerfile using <code>ENV</code>, and later switch to binding <code>/home/$USER/.aws/</code> to the container to <code>/root/.aws/</code>.</p>
  <pre><code>FROM python:3.7-alpine
  
  WORKDIR /scripts
  
  RUN pip install boto3
  
  # ENV AWS_ACCESS_KEY_ID=
  # ENV AWS_SECRET_ACCESS_KEY=
  
  COPY s3-file-upload-crontab /etc/crontabs/root
  RUN chmod 644 /etc/crontabs/root
  
  COPY s3_upload.py /scripts/s3_upload
  RUN chmod a+x /scripts/s3_upload
  
  RUN mkdir /root/info/
  RUN touch /root/info/max_mod_time.json
  RUN touch /root/info/error.log
  
  RUN mkdir /root/.aws/
  RUN touch /root/.aws/credentials
  # RUN touch /root/.aws/config
  
  
  ENTRYPOINT crond -f
  </code></pre>
  <pre><code>version: '3.8'
  services:
    s3-data-transfer:
      image: ap-aws-s3-file-upload 
      build:
        context: ./
      volumes:
        - ../data/features:/data
        - ./info:/root/info
        - ~/.aws/credentials:/root/.aws/credentials
        # - ~/.aws/config:/root/.aws/config
  </code></pre>
  <p>At this point the code is using my credentials (AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY) for authentication and works perfectly.</p>
  <p>I'm trying to switch the authentication to IAM roles. I've created a role in AWS called <code>Upload_Data_To_S3</code> with the <code>AmazonS3FullAccess</code> policy.</p>
  <p>I'm reading the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#assume-role-provider" rel="nofollow noreferrer">docs</a> on how to set up boto3 for IAM roles. I've set my <code>~/.aws/config/</code> as follows</p>
  <pre><code>[default]
  region=ca-central-1
  
  [profile crossaccount]
  role_arn=arn:aws:iam::#######:role/Upload_Data_To_S3
  source_profile=
  </code></pre>
  <p>I don't have aws cli installed so no profile, besides my user on aws account. My python code contains no code to do with authentication.</p>
  <pre class="lang-py prettyprint-override"><code>#!/usr/local/bin/python3
  
  import boto3
  from botocore.errorfactory import ClientError
  import os
  import glob
  import json
  import time
  
  # TODO: look into getting credentials from IAM role
  s3_client = boto3.client('s3')
  s3_bucket_name = 'ap-rewenables-feature-data'
  
  max_mod_time = '0'
  file_list = glob.glob('/data/*.json')  # get a list of feature files
  file_mod_time = None
  
  # get mod time for all files in data directory
  file_info = [{'file': file, 'mod_time': time.strftime(
      '%Y-%m-%d %H:%M:%S', time.gmtime(os.path.getmtime(file)))} for file in file_list]
  
  # sort files my mod time (min -&gt; max)
  timestamp_sorted_file_info = sorted(file_info, key=lambda f: f['mod_time'])
  # print('File Info Sorted by Time Stamp:\n',timestamp_sorted_file_info)
  
  # check if the file exists and not empty -&gt; set max_mod_time from it
  if os.path.exists('/root/info/max_mod_time.json') and os.stat('/root/info/max_mod_time.json').st_size != 0:
      with open('/root/info/max_mod_time.json', 'r') as mtime:
          max_mod_time = json.load(mtime)['max_mod_time']
  
  # upload the files to s3
  mod_time_last_upload = &quot;0&quot;
  for file in timestamp_sorted_file_info:
      file_mod_time = file['mod_time']  # set mod time for the current file
      # file_mod_time = '2020-09-19 13:28:53' # for debugging
      file_name = os.path.basename(file['file'])  # get file name from file path
  
      if file_mod_time &gt; max_mod_time:  # compare current file mod_time to max_mod_time from previous run
          with open(os.path.join('/data/', file_name), &quot;rb&quot;) as f:
              s3_client.upload_fileobj(f, s3_bucket_name, file_name)
  
              # error check - https://stackoverflow.com/a/38376288/7582937
              # check if the file upload was successful
              try:
                  s3_client.head_object(Bucket=s3_bucket_name, Key=file_name)
                  mod_time_last_upload = file_mod_time
                  print(file_name, ' is UPLOADED')
              except ClientError as error:
                  # Not found
                  if error.response['ResponseMetadata']['HTTPStatusCode'] == 404:
                      # save error to log file
                      open('/root/info/error.log', 'w').write(str(error))
                      print(&quot;error: &quot;, error)
                  break
  
          print('File Mod Time: ', file_mod_time)
          print('Mod Time Last Upload: ', mod_time_last_upload)
  
  
  # save max mod time to file
  # https://stackoverflow.com/a/5320889/7582937
  # create JSON object to write to the file
  object_to_write = json.dumps(
      {&quot;max_mod_time&quot;: mod_time_last_upload})
  
  # write max_mod_time to the file to be passed to the next run
  if mod_time_last_upload is not &quot;0&quot;:
      if object_to_write:
          open('/root/info/max_mod_time.json', 'w').write(str(object_to_write))
  </code></pre>
  <p>When I build and run the container I get the following error:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;/scripts/s3_upload&quot;, line 40, in &lt;module&gt;
      s3_client.upload_fileobj(f, s3_bucket_name, file_name)
    File &quot;/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py&quot;, line 539, in upload_fileobj
      return future.result()
    File &quot;/usr/local/lib/python3.7/site-packages/s3transfer/futures.py&quot;, line 106, in result
      return self._coordinator.result()
    File &quot;/usr/local/lib/python3.7/site-packages/s3transfer/futures.py&quot;, line 265, in result
      raise self._exception
    File &quot;/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py&quot;, line 126, in __call__
      return self._execute_main(kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py&quot;, line 150, in _execute_main
      return_value = self._main(**kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/s3transfer/upload.py&quot;, line 692, in _main
      client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/client.py&quot;, line 337, in _api_call
      return self._make_api_call(operation_name, kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/client.py&quot;, line 643, in _make_api_call
      operation_model, request_dict, request_context)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/client.py&quot;, line 662, in _make_request
      return self._endpoint.make_request(operation_model, request_dict)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 102, in make_request
      return self._send_request(request_dict, operation_model)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 132, in _send_request
      request = self.create_request(request_dict, operation_model)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/endpoint.py&quot;, line 116, in create_request
      operation_name=operation_model.name)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 356, in emit
      return self._emitter.emit(aliased_event_name, **kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 228, in emit
      return self._emit(event_name, kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/hooks.py&quot;, line 211, in _emit
      response = handler(**kwargs)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/signers.py&quot;, line 90, in handler
      return self.sign(operation_name, request)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/signers.py&quot;, line 160, in sign
      auth.add_auth(request)
    File &quot;/usr/local/lib/python3.7/site-packages/botocore/auth.py&quot;, line 357, in add_auth
      raise NoCredentialsError
  botocore.exceptions.NoCredentialsError: Unable to locate credentials
  </code></pre>
  <p>That's understandable since I don't have the credentials in the container. What do I need to add to the code or the <code>~/.aws/config</code> file for it to use the IAM role I've set up? Unfortunately the docs aren't very clear in this regard.</p>
  <p>Thanks in advance.</p>