Upload a directory, in parallel, to an artifact repository This code uploads an entire directory and allows for its reparenting on a remote server (an artifact repository).

Three concerns:


Copying the files locally takes a lot of space. Ideally, I could use inotify and as soon as my wget -m had a file I could begin immediately uploading and then deleting. These repositories can be hundreds of gigs
For performance reasons I fork/join everything. This definitely improves overall job performance but I get an error upload.sh: fork: retry: Resource temporarily unavailable which is the OS telling me I can't have more sub processes. My concern is that some of these processes will terminate without doing their job because of this. 
Lastly, and I can't be sure of this, but several of the larger files that upload seem to never complete when I use a fork. But when I do a 1-off they finish in an appropriate time


First I run:

wget --user cbongiorno --password abc123 --mirror --no-parent https://artifactory.stback.com/artifactory/BIF-Releases &amp;&gt;  my.logs

Then I do the upload. It would be nice to upload and delete files as they come in

#!/usr/bin/env bash

[[ -z "$1" ]] &amp;&amp; echo "no resource directory supplied" &amp;&amp; exit -1
[[ -z "$2" ]] &amp;&amp; echo "no destination repository supplied" &amp;&amp; exit -1


upload() {
    local local_repo="$1"
    local remote_repo="$2"
    local ct
    local user=${3:-${USER:-$(read -p "Username: "  user &amp;&amp; [ -n "${user}" ] &amp;&amp; echo ${user})}}
    local pwd=$(read -s -p "Password: "  pwd &amp;&amp; [ -n "${pwd}" ] &amp;&amp; echo ${pwd})
    echo ""
    declare -a pids
    for art in $(find "${local_repo}" -not -path '*/\.*' -type f -not -name 'index.html' | sed -n 's;./\(.*\);\1;p'); do


#        file extension
        case "$(echo ${art} | rev | cut -d'.' -f1 | rev)" in
            jar)
                ct="application/java-archive"
            ;;
            md5)
                ct="application/x-checksum"
            ;;
            pom)
                ct="application/x-maven-pom+xml"
            ;;
            sha1)
                ct="application/x-checksum"
            ;;
            txt)
                ct="text/plain"
            ;;
            xml)
                ct="application/xml"
            ;;
        esac
        (
            echo "${art} -&gt; ${remote_repo}/${art}"
            curl -v -u "${user}:${pwd}" -X PUT -d "@$art" -H "Content-Type: $ct" "$remote_repo/$art" 2&amp;&gt;1 &amp;&amp;
            curl -I -u "${user}:${pwd}" "$remote_repo/$art" 2&amp;&gt;1
        )  &amp;

        pids[${i}]=$!

    done
    for pid in ${pids[*]}; do
        wait ${pid}
    done
}
upload "$@"
I think you'll find that inotify has problems of its own and that you're already on the right path.  

The approach I'd use: get this working the way you want, then run it in a loop while your wget runs, removing successful files as it goes. 

If you're concerned about polling overhead (I wouldn't be: the size of the directory should remain manageable so long as you're uploading/deleting along the way), have the wget process emit filenames into a log file or a named pipe, while the curl process consumes them.

You can attack the resource problem in a couple of ways. One, use exec to launch curl so that there isn't an intermediate bash process sitting around waiting for curl to exit.  Two, keep count of the number of children, and reap them when the number gets too high.  Three, check ulimit -u and consider increasing that number.

In the code below I've made a few lesser edits:


content-types in an associative array with sensible default
use [[ ]] instead of [ ] (the former is a bash builtin)
clean up the find command line a little.  I've guessed that the sed invocation is meant to remove $local_repo?
omit forked echo and curl-HEAD; rely on exit status instead
use curl's -w to get a useful line of output on success
store pids in an associative array with filenames as values, to allow meaningful response to outcomes
"Content-type" has a lowercase t
run upload in infinite loop, keep count of files found, exit when no more files are found


There may be some bugs, since I don't have your environment to test.  Proceed accordingly.

#!/usr/bin/env bash
reap() {
    for pid in ${!pid2fn[@]}
    do
        local fn=${pid2fn[$pid]} 
        if wait ${pid} 
        then 
            echo successful upload of $fn
            # rm $fn
        else
            echo $fn upload failed with status $?         
        fi
        unset pid2fn[$pid]
    done
}
upload() {
    local local_repo="${1:?no resource directory supplied}"
    local remote_repo="${2:?no destination repository supplied}"
    local default_ct=application/octet-stream
    local user=${3:-${USER:-$(read -p "Username: "  user &amp;&amp; [[ -n $user ]] &amp;&amp; echo $user) }}
    local pwd=$(read -s -p "Password: "  pwd &amp;&amp; [[ -n $pwd ]] &amp;&amp; echo $pwd )
    local max_children=100
    local found=0
    declare -A pid2fn
    declare -A types=(
        [jar]=application/java-archive
        [md5]=application/x-checksum
        [pom]=application/x-maven-pom+xml
        [sha1]=application/x-checksum
        [txt]=text/plain
        [xml]=application/xml
    )
    for art in $( find "$local_repo" -type f -name "[^.]*" -not -name index.html | sed s,^$local_repo/*,, )
    do
        let found=found+1
        local ct=${types[${art##*.}]:-$default_ct}
        local dest="$remote_repo/$art"
        local status="$art -&gt; $dest | response %{http_code} sent %{size_upload} bytes\n"
        exec curl -u "$user:$pwd" -X PUT -d "@$art" -H "Content-type: $ct" -w "$status" "$dest" 2&gt;&amp;1 &amp;
        pid2fn[$!]=$art
        [[ ${#pid2fn[@]} -ge $max_children ]] &amp;&amp; reap
    done
    reap
    [[ $found -gt 0 ]]
}
while upload "$@"; do :; done