Getting boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request exception
Question: I am trying to **download the log files that are present in S3 bucket using
boto**. The reason behind not using s3cmd and some other tools is that I don't
want my code to be dependent on some kind of software/tool so that others can
also use my code directly and don't have to worry about downloading some other
dependencies.

I am getting the following stack trace. I saw various related posts but none
of them solved my problem.

    
    
    Traceback (most recent call last):
    File "/Library/Python/2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
    File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 405, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
    File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 171, in run
    return self.wrapped(*args, **kwargs)
    File "/pgbadger/pgbadger_html.py", line 86, in dlogs
    s3 = S3()
    File "/pgbadger/pgbadger_html.py", line 46, in __init__
    self.bucket = self._get_bucket(self.log_bucket)
    File "/pgbadger/pgbadger_html.py", line 65, in _get_bucket
    return self.s3_conn.get_bucket(bucket)
    File "/Library/Python/2.7/site-packages/boto/s3/connection.py", line 471, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
    File "/Library/Python/2.7/site-packages/boto/s3/connection.py", line 518, in head_bucket
    response.status, response.reason, body)
    boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
    

I have seen the code and I don't know why I am getting this error. My code is
as follows:

    
    
    from fabric.api import task
    from fabric.api import env
    
    S3_LOG_BUCKET = BUCKET-NAME
    
    
    class S3(object):
        s3_conn = None
        log_bucket = S3_LOG_BUCKET
        region = region
        bucket = None
        env.host_string = REGION-NAME
    
        def __init__(self):
            self._s3_connect()
            self.bucket = self._get_bucket(self.log_bucket)
    
        def _s3_connect(self):
            if not self.s3_conn:
                self.s3_conn = boto.s3.connect_to_region(
                    self.region,
                    aws_access_key_id=AWS_ACCESS_KEY_ID,
                    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                )
            if not self.s3_conn:
                raise ValueError('Invalid Region Name: {}'.format(region))
    
        def download_s3_logs(self):
            for l in self.bucket.list():
                key_string = str(l.key)
                l.get_contents_to_filename("/tempLogFiles/" + key_string)
                print l.key
    
        def _get_bucket(self, bucket):
            return self.s3_conn.get_bucket(bucket)
    
    
    @task
    def dlogs():
        s3 = S3()
        s3.download_s3_logs()
    


Answer: Problem Solved: In the S3-Log-Bucket, I was mentioning the entire path
specific to my bucket.. Like I have my bucket and there are multiple folders
in it. So I was mentioning the entire path to it but BOTO doesn't expect that
to happen. Hence I just had to mention the bucket name only instead of
mentioning the entire path.

Previously I was doing-->

    
    
    log_bucket = Bucket/Inner Folder 1/Inner Folder 2/.../ which was wrong
    

Correct way of doing it-->

    
    
    log_bucket = Bucket
    

