A temporary directory that exists only during run time of user-specified code
Always set up the trap before trying to create the directory. Otherwise there's a race condition where the script may die after creating the directory but before having a chance to clean it up. And use single quotes for the trap command to make sure the variable is only expanded at exit:

trap 'rm -rf "$tmpdir"' EXIT
tmpdir="$(mktemp -d)"


If the trap ends up being triggered before mktemp the result is simply rm -rf "", which does nothing.

Some other suggestions:


#!/usr/bin/env bash is a more portable shebang.
set -o errexit -o nounset is more readable than set -ue.
BASH_SOURCE is an array, and it's only by accident that $array_name refers to the first element of array_name. directory="$(dirname "${BASH_SOURCE[0]}")" would be more explicit.
if [[ ! "$tmpdir" || ! -d "$tmpdir" ]] is redundant. If mktemp fails the script will stop there, and if it's been removed rm -rf "$tmpdir" is safe, as mentioned.
Don't use ls in scripts!
The temporary directory can't already contain any files. That's part of the contract of mktemp - it will create a new directory if it can, or fail otherwise.