title: Transfer data from a S3 bucket to a GCP bucket using temporary credentials
body: <p>I would like to download a public dataset from the <a href="https://nda.nih.gov/" rel="nofollow noreferrer">NIMH Data Archive</a>. After creating an account on their website and accepting their Data Usage Agreement, I can download a CSV file which contains the path to all the files in the dataset I am interested in. Each path is of the form <code>s3://NDAR_Central_1/...</code>. </p>
  
  <h1>1 Download on my personal computer</h1>
  
  <p>In the <a href="https://github.com/NDAR" rel="nofollow noreferrer">NDA Github repository</a>, the <a href="https://github.com/NDAR/nda-tools" rel="nofollow noreferrer">nda-tools</a> Python library exposes some useful Python code to download those files to my own computer. Say I want to download the following file:</p>
  
  <pre><code>s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz
  </code></pre>
  
  <p>Given my username (<code>USRNAME</code>) and password (<code>PASSWD</code>) (the ones I used to create my account on the NIMH Data Archive), the following code allows me to download this file to <code>TARGET_PATH</code> on my personal computer:</p>
  
  <pre><code>from NDATools.clientscripts.downloadcmd import configure
  from NDATools.Download import Download
  
  config = configure(username=USRNAME, password=PASSWD)
  s3Download = Download(TARGET_PATH, config)
  
  target_fnames = ['s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz']
  
  s3Download.get_links('paths', target_fnames, filters=None)
  s3Download.get_tokens()
  s3Download.start_workers(False, None, 1)
  </code></pre>
  
  <p>Behind the hood, the <a href="https://github.com/NDAR/nda-tools/blob/50f9cb15da8988aeed49709652d3a08343eae7ea/NDATools/Download.py#L200" rel="nofollow noreferrer"><code>get_tokens</code></a> method of <code>s3Download</code> will use <code>USRNAME</code> and <code>PASSWD</code> to generate <strong>temporary</strong> access key, secret key and security token. Then, the <a href="https://github.com/NDAR/nda-tools/blob/50f9cb15da8988aeed49709652d3a08343eae7ea/NDATools/Download.py#L266" rel="nofollow noreferrer"><code>start_workers</code></a> method will use the boto3 and s3transfer Python libraries to <a href="https://github.com/NDAR/nda-tools/blob/50f9cb15da8988aeed49709652d3a08343eae7ea/NDATools/Download.py#L244" rel="nofollow noreferrer">download</a> the selected file.</p>
  
  <p>Everything works fine !</p>
  
  <h1>2 Download to a GCP bucket</h1>
  
  <p>Now, say I created a project on GCP and would like to directly download this file to a GCP bucket.</p>
  
  <p>Ideally, I would like to do something like:</p>
  
  <pre><code>gsutil cp s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz gs://my-bucket
  </code></pre>
  
  <p>To do this, I execute the following Python code in the Cloud Shell (by running <code>python3</code>):</p>
  
  <pre><code>from NDATools.TokenGenerator import NDATokenGenerator
  data_api_url = 'https://nda.nih.gov/DataManager/dataManager'
  generator = NDATokenGenerator(data_api_url)
  token = generator.generate_token(USRNAME, PASSWD)
  </code></pre>
  
  <p>This gives me the access key, the secret key and the session token. Indeed, in the following, </p>
  
  <ul>
  <li><code>ACCESS_KEY</code> refers to the value of <code>token.access_key</code>,</li>
  <li><code>SECRET_KEY</code> refers to the value of <code>token.secret_key</code>,</li>
  <li><code>SECURITY_TOKEN</code> refers to the value of <code>token.session</code>.</li>
  </ul>
  
  <p>Then, I set these credentials as environment variables in the Cloud Shell:</p>
  
  <pre><code>export AWS_ACCESS_KEY_ID = [copy-paste ACCESS_KEY here]
  export AWS_SECRET_ACCESS_KEY = [copy-paste SECRET_KEY here]
  export AWS_SECURITY_TOKEN = [copy-paste SECURITY_TOKEN here]
  </code></pre>
  
  <p>Eventually, I also set up the <code>.boto</code> configuration file in my home. It looks like this:</p>
  
  <pre><code>[Credentials]
  aws_access_key_id = $AWS_ACCESS_KEY_ID
  aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
  aws_session_token = $AWS_SECURITY_TOKEN
  [s3]
  calling_format = boto.s3.connection.OrdinaryCallingFormat
  use-sigv4=True
  host=s3.us-east-1.amazonaws.com
  </code></pre>
  
  <p>When I run the following command:</p>
  
  <pre><code>gsutil cp s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz gs://my-bucket
  </code></pre>
  
  <p>I end up with:</p>
  
  <blockquote>
    <p>AccessDeniedException: 403 AccessDenied</p>
  </blockquote>
  
  <p>The full traceback is below:</p>
  
  <pre><code>Non-MD5 etag ("a21a0b2eba27a0a32a26a6b30f3cb060-6") present for key &lt;Key: NDAR_Central_1,submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz&gt;, data integrity checks are not possible.
  Copying s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz [Content-Type=application/x-gzip]...
  Exception in thread Thread-2:iB]
  Traceback (most recent call last):
    File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
      self.run()
    File "/usr/lib/python2.7/threading.py", line 754, in run
      self.__target(*self.__args, **self.__kwargs)
    File "/google/google-cloud-sdk/platform/gsutil/gslib/daisy_chain_wrapper.py", line 213, in PerformDownload
      decryption_tuple=self.decryption_tuple)
    File "/google/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 353, in GetObjectMedia
      decryption_tuple=decryption_tuple)
    File "/google/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 590, in GetObjectMedia
      generation=generation)
    File "/google/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 1723, in _TranslateExceptionAndRaise
      raise translated_exception  # pylint: disable=raising-bad-type
  AccessDeniedException: AccessDeniedException: 403 AccessDenied
  &lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access Denied&lt;/Message&gt;&lt;RequestId&gt;A93DBEA60B68E04D&lt;/RequestId&gt;&lt;HostId&gt;Z5XqPBmUdq05btXgZ2Tt7HQMzodgal6XxTD6OLQ2sGjbP20AyZ+fVFjbNfOF5+Bdy6RuXGSOzVs=&lt;/HostId&gt;&lt;/Error&gt;
  
  AccessDeniedException: 403 AccessDenied
  &lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access Denied&lt;/Message&gt;&lt;RequestId&gt;A93DBEA60B68E04D&lt;/RequestId&gt;&lt;HostId&gt;Z5XqPBmUdq05btXgZ2Tt7HQMzodgal6XxTD6OLQ2sGjbP20AyZ+fVFjbNfOF5+Bdy6RuXGSOzVs=&lt;/HostId&gt;&lt;/Error&gt;
  </code></pre>
  
  <blockquote>
    <p>I would like to be able to directly download this file from a S3 bucket to my GCP bucket (without having to create a VM, setup Python and run the code above [which works]). Why is it that the temporary generated credentials work on my computer but do not work in GCP Cloud Shell?</p>
  </blockquote>
  
  <p>The complete log of the debug command</p>
  
  <pre><code>gsutil -DD cp s3://NDAR_Central_1/submission_13364/00m/0.C.2/9007827/20041006/10263603.tar.gz gs://my-bucket
  </code></pre>
  
  <p>can be found <a href="https://www.dropbox.com/s/2khp53pmej00vkc/complete_debug_log.txt?dl=0" rel="nofollow noreferrer">here</a>.</p>