Inverse z-transform of a modulus square
What you're probably looking for is the inverse (discrete Fourier) transform of $|X(e^{j\omega})|^2$, i.e., of $|X(z)|^2$ for $|z|=1$. Otherwise, as pointed out in Fat32's answer, $|X(z)|^2$ is generally no valid $\mathcal{Z}$-transform.

Note that a valid $\mathcal{Z}$-transform must have the form of a power series:

$$X(z)=\sum_{n=-\infty}^{\infty}x[n]z^{-n}\tag{1}$$

Take as a simple example the $\mathcal{Z}$-transform of $x[n]=\delta[n]+\delta[n-1]$:

$$X(z)=1+z^{-1}\tag{2}$$

Then $|X(z)|^2$ is given by

$$|X(z)|^2=\left|1+z^{-1}\right|^2=1+2\text{Re}\{z^{-1}\}+\left|z^{-1}\right|^2\tag{3}$$

which is not of the form $(1)$, hence it is no valid $\mathcal{Z}$-transform.

Since $|X(e^{j\omega})|^2=X(e^{j\omega})X^*(e^{j\omega})$, you need to find the IDFT of $X^*(e^{j\omega})$:

$$\frac{1}{2\pi}\int_{-\pi}^{\pi}X^*(e^{j\omega})e^{jn\omega}d\omega=\frac{1}{2\pi}\left[\int_{-\pi}^{\pi}X(e^{j\omega})e^{-jn\omega}d\omega\right]^*=x^*[-n]\tag{4}$$

Consequently, the IDFT of $|X(e^{j\omega})|^2$ is given by $x[n]\star x^*[-n]$, where $\star$ denotes convolution.

It's also good to know what this means in terms of the $\mathcal{Z}$-transform. The $\mathcal{Z}$-transform of $x^*[-n]$ can be derived as follows:

$$\sum_nx^*[-n]z^{-n}=\sum_nx^*[n]\left(\frac{1}{z}\right)^{-n}=\left[\sum_nx[n]\left(\frac{1}{z^*}\right)^{-n}\right]^*=X^*\left(\frac{1}{z^*}\right)\tag{5}$$

So the $\mathcal{Z}$-transform equivalent of $|X(e^{j\omega})|^2$ is $X(z)X^*(1/z^*)$, as they correspond to the same time domain sequence. As shown above, the function $X(z)X^*(1/z^*)$ is a valid $\mathcal{Z}$-transform, unlike $|X(z)|^2$.