Crawl and gather all the urls recursively in a domain This is the script :

#!/bin/bash

set -euo pipefail

SOFT="$(basename $0)"
random=$(( ( RANDOM % 100000 ) ))
tempfolder="/tmp/$SOFT/$random"

mkdir -p "$tempfolder"
pushd "$tempfolder" &gt; /dev/null
cleanup (){
    cat wget.txt | sed 's/:$//'  | url_matcher
    popd &gt; /dev/null
    rm -rf -- ${tempfolder}
}

trap cleanup EXIT
site="$1"

wget -nd --spider --recursive --no-verbose --output-file=wget.txt "$site"


the script essentially creates a subfolder in /tmp/crawl and puts into it the result of wget, then feeds it to sed and url_matcher.

Initially I though of putting the cat instruction just after the wget one, but for long wget it didn't work.

The sed instruction removes colons at the end of lines, which is added by wget when said url isn't valid.
(something along the lines of "http://...: Nothing there")

url_matcher is a c++ scanner made with flex, which recognises urls in a text fed with standard input and prints them to standard output, separated with new lines characters.

the scripts allows me to fetch all urls in a website and use that as a stream directly.

Example : 

$ crawl www.example.com | sed 's/some_treatments//' &gt; super_file
There's no need to use cat here:
cat wget.txt | sed 's/:$//'  | url_matcher

Instead, simply have sed take its input from the file:
sed 's/:$//' wget.txt | url_matcher


Instead of using $RANDOM and mkdir, we can use the mktemp command (GNU coreutils) to create us a new directory (without risk of conflict with an existing one):
tempfolder=$(mktemp -d)

I think that now removes the only Bashism in the script.

Avoid pushd and popd in scripts (having to redirect the unwanted output should be a sign that these are intended for interactive shells).  Instead, we can use cd &quot;$tempfolder&quot; and something like cd .. to leave it (it doesn't matter where we end up, as we only use absolute paths after that).  Alternatively, don't change directory at all, but specify paths in full (I'll use a shorter name for the temporary directory):
dir=$(mktemp -d)

cleanup() {
    sed -e 's/:$//' &quot;$dir/wget.txt&quot; | url_matcher
    rm -rf -- &quot;$dir&quot;
}

trap cleanup EXIT

wget -nd --spider --recursive --no-verbose --output-file=&quot;$dir/wget.txt&quot; &quot;$1&quot;

Other alternatives to pushd/popd include saving $PWD into a variable before changing directory, or using cd - (which internally does exactly that, as cd saves $PWD into $OLDPWD for that purpose).

We can avoid the temporary file altogether, to simplify the code and to allow us to start producing output without waiting for the crawl to finish.  We can do this in a couple of ways, by being creative with --output-file.  If we're on Linux, we can write to /dev/stdout instead of a file:
wget -nd --spider --recursive --no-verbose --output-file=/dev/stdout &quot;$1&quot; \
  | sed -e 's/:$//' | url_matcher

If we're happy to accept a Bash dependency again, we can use process substitution:
wget -nd --spider --recursive --no-verbose \
     --output-file=&gt;(sed -e 's/:$//' | url_matcher) &quot;$1&quot;

However, neither of those dependencies are necessary, as wget follows the common convention of treating the filename - specially, to mean &quot;standard output stream&quot;:
wget -nd --spider --recursive --no-verbose --output-file=- &quot;$1&quot; \
  | sed -e 's/:$//' | url_matcher


Modified code
#!/bin/bash

set -uo pipefail

exec wget -nd --spider --recursive --no-verbose --output-file=- &quot;$1&quot; \
    | sed -e 's/:$//' | url_matcher