Parallel download of rpm packages I use zypper to download and install for openSUSE Tumbleweed (much like everyone else of course). Unfortunately it downloads packages serially, and having about a thousand packages to update a week, it gets quite boring. Also, while zypper can download packages one-by-one in advance, it can't be called concurrently. I found the libzypp-bindings project but it is discontinued. I set myself to improve the situation.

Goals:


Download all repositories in parallel (most often different servers);
Download up to MAX_PROC (=6) packages from each repository in parallel;
Save packages where zypper picks them up during system update: /var/cache/zypp/packages;
Alternatively, download to $HOME/.cache/zypp/packages;
Avoid external dependencies, unless necessary.


Outline:


Find the list of packages to update;
Find the list of repositories;
For each repository:


Keep up to $MAX_PROC curl processes downloading packages.

Copy files to default package cache;


#!/bin/bash

MAX_PROC=6

function repos_to_update () {
    zypper list-updates | grep '^v ' | awk -F '|' '{ print $2 }' | sort --unique | tr -d ' '
}

function packages_from_repo () {
    local repo=$1

    zypper list-updates | grep " | $repo " | awk -F '|' '{ print $6, "#", $3, "-", $5, ".", $6, ".rpm" }' | tr -d ' '
}

function repo_uri () {
    local repo=$1

    zypper repos --uri | grep " | $repo " | awk -F '|' '{ print $7 }' | tr -d ' '
}

function repo_alias () {
    local repo=$1

    zypper repos | grep " | $repo " | awk -F '|' '{ print $2 }' | tr -d ' '
}

function download_package () {
    local alias=$1
    local uri=$2
    local line=$3
    IFS=# read arch package_name &lt;&lt;&lt; "$line"

    local package_uri="$uri/$arch/$package_name"
    local local_dir="$HOME/.cache/zypp/packages/$alias/$arch"
    local local_path="$local_dir/$package_name"
    printf -v y %-30s "$repo"
    printf "Repository: $y Package: $package_name\n"
    if [ ! -f "$local_path" ]; then
        mkdir -p $local_dir
        curl --silent --fail -L -o $local_path $package_uri
    fi
}

function download_repo () {
    local repo=$1

    local uri=$(repo_uri $repo)
    local alias=$(repo_alias $repo)
    local pkgs=$(packages_from_repo $repo)
    local max_proc=$MAX_PROC
    while IFS= read -r line; do
        if [ $max_proc -eq 0 ]; then
            wait -n
            ((max_proc++))
        fi
        download_package "$alias" "$uri" "$line" &amp;
        ((max_proc--))
    done &lt;&lt;&lt; "$pkgs"
    wait
}

function download_all () {
    local repos=$(repos_to_update)
    while IFS= read -r line; do
        download_repo $line &amp;
    done &lt;&lt;&lt; "$repos"
    wait
}

download_all
sudo cp -r ~/.cache/zypp/packages/* /var/cache/zypp/packages/


There's 2 or 3 places where grep/tr are subject to issues, but the nature of the data doesn't require much more than that.
I completely changed the approach which resulted in faster downloads, and improved functionality. The overall approach of the initial version works well for a generic solution looking for adding concurrency to independent jobs.


Removed zypper list-updates. This format is easier for machine consumption, but it's not intended for Tumbleweed. Replaced with zypper dup/inr/in --details;
Removed background jobs and waits. Replaced with aria2 which handles the maximum number of concurrent connections;
It pays off learning more about awk capabilities, in order to replace sequences of grep/awk/tr with a single awk;
The main job of the script became building a plain text file with URIs and target directory for each .rpm file


aria2 is really great tool. Superb quality. curl is not reliable in its native concurrent download capabilities.