What does $(\mathbb{Z}_n^*)^2$ mean? In a paper they write once, $(\mathbb{Z}_n^*)^2$. Is this the group of quadratic residues or is it something else?

Here the theorem:

Under the strong RSA assumption, given a modulus $n$, along with random elements $g, h \in (\mathbb{Z}_n^∗)^2$, it is hard to compute $w \in \mathbb{Z}_n^*$ and integers $a, b, c$ such that $w^c = g^a h^b$ and ($c \nmid a$ or $c \nmid b$).
$\mathbb Z_n^*$ is a mathematical notation for the multiplicative group of integers modulo $n$.  In other words, it is the set of integers that are relatively prime to $n$, all taken modulo $n$ (excluding zero). The $*$ symbol is commonly used for denoting "the set of elements in the multiplicative group", which in this case means "the set of elements that are relatively prime to $n$". 

Normally $\mathbb Z_n^*$ is treated as a multiplicative group, so it comes coupled with the multiplication operation modulo $n$.  (In contrast $\mathbb Z_n$ would denote the group of integers modulo $n$ under addition, or the field over the same set with both addition and multiplication.)

When such algebraic structures are raised to the power of two, it is an abbreviation for the Cartesian product of the underlying set with itself, in this case $\mathbb Z_n^* \times\mathbb Z_n^*$, i.e. the set of value pairs $(x,y)$ where both $x,y$ are integers in the range $1$ to $n-1$ and relatively prime to $n$.

Cartesian products of sets are also sets and might be coupled with one or more binary operations to form algebraic structures, such as fields and groups. In this case, the operations would be determined by context.

Edit: It should be noted that in mathematics, the $*$ symbol is more generally used for denoting a set excluding element zero. For instance, $\mathbb Z^*$ means the set of integers except $0$. However, it doesn't make sense to extrapolate this notation to finite sets such as $\mathbb Z_n^*$, except when $n$ is prime. If $n$ is composite, the set of integers $1$ to $n-1$ is not closed under multiplication, but the set of integers relatively prime to $n$ is. Hence, it is usually safe to presume $\mathbb Z_n^*$ denotes the (underlying set of the) multiplicative subgroup, unless explicitly stated otherwise.