Eucalyptus Walrus/Amazon S3 SOAP signature is failing
Question: I have been learning how to use Amazon S3 API by using the open source
Eucalyptus. So far I have been able to successfully use REST, but now I would
also like to use SOAP. I seem to be having trouble generating the correct
signature for my request. The service is giving me a 403 Forbidden error:

    
    
    Traceback (most recent call last):
      File "soap.py", line 31, in <module>
        r = w.download_file('mybucket', 'test.txt')
      File "soap.py", line 27, in download_file
        r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
      File "/usr/lib/python2.6/site-packages/suds/client.py", line 521, in __call__
        return client.invoke(args, kwargs)
      File "/usr/lib/python2.6/site-packages/suds/client.py", line 581, in invoke
        result = self.send(soapenv)
      File "/usr/lib/python2.6/site-packages/suds/client.py", line 619, in send
        description=tostr(e), original_soapenv=original_soapenv)
      File "/usr/lib/python2.6/site-packages/suds/client.py", line 677, in process_reply
        raise Exception((status, description))
    Exception: (403, u'Forbidden')
    

My code is in Python 2 and uses the SUDS-Jurko library for sending SOAP
requests:

    
    
    from suds.client import Client
    
    class WalrusSoap:
        wsdl_url = 'https://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl'
        server_url = 'https://localhost:8773/services/Walrus'
    
        def __init__(self, access_key, secret_key):
            self.access_key = access_key
            self.secret_key = secret_key
            self.client = Client(self.wsdl_url)
            self.client.wsdl.services[0].setlocation(self.server_url)
            #print self.client
    
        def create_signature(self, operation, timestamp):
            import base64, hmac, hashlib
            h = hashlib.sha1(self.secret_key)
            h.update("AmazonS3" + operation + timestamp)
            #h = hmac.new(key=self.secret_key, msg="AmazonS3" + operation + timestamp, digestmod=hashlib.sha1)
            return base64.encodestring(h.digest()).strip()
    
        def download_file(self, bucket, filename):
            from time import gmtime, strftime
            timestamp = strftime('%Y-%m-%dT%H:%M:%S.001Z', gmtime())
            print(timestamp)
            signature = self.create_signature('ListAllMyBuckets', timestamp)
            print(signature)
            r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
            return r
    
    w = WalrusSoap(access_key='MOBSE7FNS6OC5NYC75PG8', secret_key='yxYZmSLCg5Xw6rQVgoIuVLMAx3hZRlxDc0VOJqox')
    r = w.download_file('mybucket', 'test.txt')
    print(r)
    

I changed the server endpoint, because otherwise the WSDL points to the
regular S3 servers at Amazon. I also have two different ways of creating the
signature in my create_signature function. I was swapping between one and the
other by simply commenting out the second one. Neither of the two seem to
work. My question is what am I doing wrong?

SOAP Authentication:
<http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAuthentication.html>

SUDS-Jurko Documentation: <https://bitbucket.org/jurko/suds/overview>

Edit: I realized I forgot to include an example of what timestamp and
signature is printed for debugging purposes.

    
    
    2014-12-05T00:27:41.001Z
    0h8vxE2+k10tetXZQJxXNnNUjjw=
    

Edit 2: Also, I know that the download_file function does not download a file
:) I am still in testing/debug phase

Edit 3: I am aware that REST is better to use, at least according to Amazon.
(Personally I think REST is better also.) I am also already aware that SOAP is
deprecated by Amazon. However I would like to go down this path anyways, so
please do me a favor and do not waste my time with links to the deprecation. I
assure you that while writing this SOAP code, I was already well aware of the
deprecation. In fact one of the links I posted has the deprecation notice
printed at the top of its page. However, if you have evidence showing that
Walrus completely ditches SOAP or that they stopped working on the SOAP
portion, I would like to see something like that. But please do not tell me
Amazon has deprecated SOAP.


Answer: The S3 SOAP API does not support "new" features so the REST API should be used
where possible:

<http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAPI3.html>

<https://forums.aws.amazon.com/message.jspa?messageID=77821>

IIRC recent versions of Eucalyptus do not support SOAP with S3.

That said, the signature looks good to me so I would check if the
client/service hosts have the correct time, if there is a difference of more
than 15 minutes authentication would fail.

You could also check the cloud-error.log on the Walrus service host as there
may be more details on the failure there.

