Extracting large tar.bz2 folders in parallel The bash below utilizes parallel to extract tar.bz2 folders with 9 files in them. The total size of each is around 60GB. Below are some specs of my system. My question is the below the best/fastest (currently takes ~ 1 hour) way to extract these large tar.bz2 folders? I do not want to slow my system down or damage it so I wanted to ask more knowledgable users.


OS = Ubuntu 14.04, 64 bit
Memory = 128 GB
Processor = Intel® Xeon(R) CPU E5-2630 v3 @ 2.40GHz × 16




#!/bin/bash

# parallel untar of tar.bz2

maxproc=3 # Max number of threads.
i=0

# Count files
set -- /path/to/folders/*.tar.bz2
FILES="$#"

# Blank $1 $2 ...
set --

let i=1
for FILE in /path/to/folders/*.tar.bz2
do
    printf "(%2d/%2d)\tProcessing %s\n" "$i" "$FILES" "$FILE"
    let i=i+1

    tar -xvjf "$FILE" -C /path/to/folders &gt;/dev/null &amp;

    # Turn $1=pida $2=pidb $3=pidc $4=pidd, into
    #      $1=pida $2=pidb $3=pidc $4=pidd $5=pide
    set -- "$@" $!

    # Shift removes $1 and moves the rest down, so you get
    # $1=pidb $2=pidc $3=pidd $3=pide
    # $# is the number of arguments.
    if [ "$#" -ge $maxproc ]  ; then wait "$1" ; shift; fi
done

wait
Re-purposing the argument list this way is not idiomatic Bash, and it took me some time to realise that (firstly) the supplied arguments are ignored and replaced with a list from filesystem and that (secondly) those arguments are replaced with the process IDs of the child tar processes.  It might be better to use distinctly-named array variables instead.



There's very little point in passing the -v argument to tar and then redirecting it to /dev/null.  If you don't pass -v, then tar will quietly get on with its job:

    tar -xjf "$FILE" -C /path/to/folders &amp;




Why is maxproc set to 3?  Is that an empirical value based on your observed disk (and filesystem) throughput?  If so, provide a means to overwrite it (e.g. allow it to be passed as an environment variable).

If you want to spawn one child for each processor (core) on your system, you might choose maxproc=$(getconf _NPROCESSORS_ONLN); I'll leave it as an exercise to think of other possible defaults.



The wait command here waits for a specific process to exit.  If job 2 finishes before job 1, then the script won't start a new process to replace it until after job 1 completes, resulting in lower overall utilisation, especially if the jobs are of unequal size or proceeding at unequal rates.

Instead, replacing wait "$1" with wait -n will wait for any one child process to exit.  This is more likely to be what we want.



The script always returns a success value, even if one or more extractions fail (e.g. due to a full filesystem).  It's possible to remember that an error occurred:

status=0
for FILE in ...
do
    ...
    wait -n || status=1
done

wait || status=1
exit $status




Consider using an existing implementation of parallel or xjobs rather than reinventing the wheel.  That turns this script into a simple one-liner.