Crawl and gather all the urls recursively in a domain
There's no need to use cat here:
cat wget.txt | sed 's/:$//'  | url_matcher

Instead, simply have sed take its input from the file:
sed 's/:$//' wget.txt | url_matcher


Instead of using $RANDOM and mkdir, we can use the mktemp command (GNU coreutils) to create us a new directory (without risk of conflict with an existing one):
tempfolder=$(mktemp -d)

I think that now removes the only Bashism in the script.

Avoid pushd and popd in scripts (having to redirect the unwanted output should be a sign that these are intended for interactive shells).  Instead, we can use cd &quot;$tempfolder&quot; and something like cd .. to leave it (it doesn't matter where we end up, as we only use absolute paths after that).  Alternatively, don't change directory at all, but specify paths in full (I'll use a shorter name for the temporary directory):
dir=$(mktemp -d)

cleanup() {
    sed -e 's/:$//' &quot;$dir/wget.txt&quot; | url_matcher
    rm -rf -- &quot;$dir&quot;
}

trap cleanup EXIT

wget -nd --spider --recursive --no-verbose --output-file=&quot;$dir/wget.txt&quot; &quot;$1&quot;

Other alternatives to pushd/popd include saving $PWD into a variable before changing directory, or using cd - (which internally does exactly that, as cd saves $PWD into $OLDPWD for that purpose).

We can avoid the temporary file altogether, to simplify the code and to allow us to start producing output without waiting for the crawl to finish.  We can do this in a couple of ways, by being creative with --output-file.  If we're on Linux, we can write to /dev/stdout instead of a file:
wget -nd --spider --recursive --no-verbose --output-file=/dev/stdout &quot;$1&quot; \
  | sed -e 's/:$//' | url_matcher

If we're happy to accept a Bash dependency again, we can use process substitution:
wget -nd --spider --recursive --no-verbose \
     --output-file=&gt;(sed -e 's/:$//' | url_matcher) &quot;$1&quot;

However, neither of those dependencies are necessary, as wget follows the common convention of treating the filename - specially, to mean &quot;standard output stream&quot;:
wget -nd --spider --recursive --no-verbose --output-file=- &quot;$1&quot; \
  | sed -e 's/:$//' | url_matcher


Modified code
#!/bin/bash

set -uo pipefail

exec wget -nd --spider --recursive --no-verbose --output-file=- &quot;$1&quot; \
    | sed -e 's/:$//' | url_matcher