Boto3 to download all files from a S3 Bucket
Question: I'm using boto3 to get files from s3 bucket. I need a similar functionality
like `aws s3 sync`

My current code is

    
    
    #!/usr/bin/python
    import boto3
    s3=boto3.client('s3')
    list=s3.list_objects(Bucket='my_bucket_name')['Contents']
    for key in list:
        s3.download_file('my_bucket_name', key['Key'], key['Key'])
    

This is working fine, as long as the bucket has only files. If a folder is
present inside the bucket, its throwing an error

    
    
    Traceback (most recent call last):
      File "./test", line 6, in <module>
        s3.download_file('my_bucket_name', key['Key'], key['Key'])
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/inject.py", line 58, in download_file
        extra_args=ExtraArgs, callback=Callback)
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 651, in download_file
        extra_args, callback)
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 666, in _download_file
        self._get_object(bucket, key, filename, extra_args, callback)
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 690, in _get_object
        extra_args, callback)
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 707, in _do_get_object
        with self._osutil.open(filename, 'wb') as f:
      File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 323, in open
        return open(filename, mode)
    IOError: [Errno 2] No such file or directory: 'my_folder/.8Df54234'
    

Is this a proper way to download a complete s3 bucket using boto3. How to
download folders.


Answer: I got the same needs and create the following function that download
recursively the files. The directories are created locally only if they
contain files.

    
    
    import boto3
    import os
    
    def download_dir(client, resource, dist, local='/tmp', bucket='your_bucket'):
        paginator = client.get_paginator('list_objects')
        for result in paginator.paginate(Bucket=bucket, Delimiter='/', Prefix=dist):
            if result.get('CommonPrefixes') is not None:
                for subdir in result.get('CommonPrefixes'):
                    download_dir(client, resource, subdir.get('Prefix'), local)
            if result.get('Contents') is not None:
                for file in result.get('Contents'):
                    if not os.path.exists(os.path.dirname(local + os.sep + file.get('Key'))):
                         os.makedirs(os.path.dirname(local + os.sep + file.get('Key')))
                    resource.meta.client.download_file(bucket, file.get('Key'), local + os.sep + file.get('Key'))
    

The function is called that way:

    
    
    def _start():
        client = boto3.client('s3')
        resource = boto3.resource('s3')
        download_dir(client, resource, 'clientconf/', '/tmp')
    

