S3 Bash Tools Part 3 This is a follow on to a previous question:
Part 3: User Scripts
All the code I am submitting here is available in github

The reason to write these tools is so I can prototype and quickly verify that I am doing things correctly before implementing them in C++. This is mainly necessary because the documentation these scripts are based on are relatively bad and confusing.

The scripts used by users:
s3cp
#!/usr/bin/env bash

badflag=0
declare -a ARGS

function usage {
    echo &quot;Usage:&quot;
    echo &quot;s3cp  &lt;file&gt; &lt;s3-url&gt; [--key=&lt;key&gt;] [--secret=&lt;secret&gt;] [--verbose] [--dryrun]&quot;
    echo &quot;s3cp  &lt;s3-url&gt; &lt;file&gt; [--key=&lt;key&gt;] [--secret=&lt;secret&gt;] [--verbose] [--dryrun]&quot;
    echo
    echo &quot;s3-url:  s3://&lt;bucket&gt;/&lt;Object-Path&gt;&quot;
    exit 1
}

verbose=''
dryrun=0
access=${AMZ_ACCESS_KEY}
secret=${AMZ_SECRET_KEY}
for var in &quot;$@&quot;; do
    if [[ &quot;${var}&quot; == '--verbose' ]]; then
        verbose='/dev/stdout'
        continue
    fi
    if [[ &quot;${var}&quot; == '--dryrun' ]]; then
        dryrun=1
        continue;
    fi
    if [[ &quot;${var%=*}&quot; == &quot;--key&quot; ]]; then
        access=${var#*=}
        continue;
    fi
    if [[ &quot;${var%=*}&quot; == &quot;--secret&quot; ]]; then
        secret=${var#*=}
        continue;
    fi
    flag=${var#-}
    if [[ &quot;${flag}&quot; != ${var} ]]; then
        badflag=1
        echo &quot;Bad flag: ${var}&quot;
        continue
    fi
    ARGS+=(&quot;$var&quot;)
done

if [[ &quot;${access}&quot; == &quot;&quot; || &quot;${secret}&quot; == &quot;&quot; ]]; then
    echo &quot;Error: No Access or Secret provided.&quot;
    echo &quot;These can be set in the environment variables AMZ_ACCESS_KEY/AMZ_SECRET_KEY or passed with flags&quot;
    usage
fi

if [[ ${#ARGS[@]} != 2 || ${badflag} != 0 ]]; then
    usage
fi

src=${ARGS[0]}
dst=${ARGS[1]}

doUpload=0
doDownload=0
srcS3=${src#s3://}
if [[ &quot;${src}&quot; == &quot;${srcS3}&quot; ]]; then
    doUpload=1
fi
dstS3=${dst#s3://}
if [[ &quot;${dst}&quot; == &quot;${dstS3}&quot; ]]; then
    doDownload=1
fi

if [[ ${doUpload} == 0 &amp;&amp; ${doDownload} == 0 ]]; then
    echo &quot;Neither source or destination is on S3. Just use normal cp&quot;
    usage
fi
if [[ ${doUpload} == 1 &amp;&amp; ${doDownload} == 1 ]]; then
    echo &quot;Both source or destination is on S3. Don't support move yet.&quot;
    usage
fi

dir=&quot;$(cd &quot;$(dirname &quot;${BASH_SOURCE[0]}&quot;)&quot; &amp;&amp; pwd)&quot;
source ${dir}/s3/signature


if [[ ${doUpload} == 1 ]]; then
    source ${dir}/s3/upload
    upload &quot;${src}&quot; &quot;${dst}&quot; &quot;${access}&quot; &quot;${secret}&quot; '' &quot;${verbose}&quot; &quot;${dryrun}&quot;
else
    source ${dir}/s3/download
    download &quot;${src}&quot; &quot;${dst}&quot; &quot;${access}&quot; &quot;${secret}&quot; '' &quot;${verbose}&quot; &quot;${dryrun}&quot;
fi
In case of errors/warnings, write your output to stderr and not just echo them to stdout stream. This might also include you returning the usage section.

Having a chain of if block is imo, always very bad. You can use a switch-case block there, and keep shifting the arguments. A simple example would be:

ARGS=()
while [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
      -v|--verbose)
        verbose='/dev/stdout'
        shift # past argument
        ;;
      --dryrun)
        dryrun=1
        shift # past argument
        ;;
      -k=*|--key=*)
        access="${key#*=}"
        shift # past argument
        shift # past value
        ;;
      -k|--key)
        shift # past argument
        access="$1"
        shift # past value
        ;;
      .
      .
      .
      *)    # unknown option
        if [[ "${key#-}" != ${key} ]]; then
            badflag=1
            &gt;&amp;2 echo "Bad flag: ${var}"
        fi
        ARGS+=("$1") # save it in an array for later
        shift # past argument
        ;;
    esac
done


Notice how easily you can maintain the --key=KEY and --key KEY format parameters. Apart from that, it is really easy to later on add another new argument to your utility.

Keep defining your globals in UPPER_CASE, so that it easy to distinguish them later in your ever-expanding code.


echo "These can be set in the environment variables AMZ_ACCESS_KEY/AMZ_SECRET_KEY or passed with flags"



AWS recommends having the environment variables named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. I would consider it breaking the norm if you are not respecting defaults.

Define the usage content (or man page?) as a heredoc:

USAGE=&lt;&lt;EOF

Usage:
s3cp  &lt;file&gt; &lt;s3-url&gt; [--key=&lt;key&gt;] [--secret=&lt;secret&gt;] [--verbose] [--dryrun]
s3cp  &lt;s3-url&gt; &lt;file&gt; [--key=&lt;key&gt;] [--secret=&lt;secret&gt;] [--verbose] [--dryrun]

s3-url:  s3://&lt;bucket&gt;/&lt;Object-Path&gt;

EOF