title: AWS S3: Remove Object Prefix From Thousands of Files in Complex Directory Structure
body: <p>I am using AWS CLI interface to manage files/objects in S3.  I have thousands of objects buried in a complex system of nested folders (subfolders), I want to elevate all of the objects to the “root” of the S3 bucket, in one folder at the root of the bucket <code>(s3://bucket/folder/file.txt)</code>.</p>
  
  <p>I've tried using this command:</p>
  
  <pre><code> aws s3 s3://bucket-a/folder-a s3://bucket-a --recursive --exclude “*” --include “*.txt”
  </code></pre>
  
  <p>When I use the mv command, it carries over the prefixes (directory paths) of each object resulting in the same nested folder system.  Here is what I want to accomplish:</p>
  
  <p>Desired Result:
  Where:</p>
  
  <pre><code>s3://bucket-a/folder-a/file-1.txt
  s3://bucket-a/folder-b/folder-b1/file-2.txt
  s3://bucket-a/folder-c/folder-c1/folder-c2/ file-3.txt
  </code></pre>
  
  <p>Output:
  s3://bucket-a/file-1.txt
  s3://bucket-a/file-2.txt
  s3://bucket-a/file-3.txt</p>
  
  <p>I have been told, that I need to use a bash script to accomplish my desired result.  Here is a sample script that was provided to me:</p>
  
  <pre><code>#!/bin/bash
  #BASH Script to move objects without directory structure
  bucketname='my-bucket'
  for key in $(aws s3api list-objects --bucket "${my-bucket}" --query "Contents[].{Object:Key}" --output text) ;
    do
      echo "$key"
      FILENAME=$($key | awk '{print $NF}' FS=/)
      aws s3 cp s3://$my-bucket/$key s3://$my-bucket/my-folder/$FILENAME
    done
  </code></pre>
  
  <p>When I run this bash script, I get an error:</p>
  
  <blockquote>
    <p>A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied</p>
  </blockquote>
  
  <p>I tested the connection with another aws s3 command and confirmed that it works.  I added policies to the user to include all privledges to s3, I have no idea what I am doing wrong here.</p>
  
  <p>Any help would be greatly appreciated.</p>