Is $y[n]=x[n] * x[n^2]$ invertible?
The system is nonlinear (bilinear in $x$), with a nonlinear law (square) on indices. Odds are the system is not invertible. One can try to prove it in its full generally, or try to find a counterexample. Let us try the lazy way, using properties at hand. 

The system is bilinear. Hence, the output for $k x[n]$, $k\in \mathbb{R} $, will be  $k^2 y[n]$. Resultingly, since $(-k)^2 = k^2$,  $k x[n]$ and  $-k x[n]$ have the same output. If they are nonzero, they differ and yield the same result. Verdict: non-invertible.

This is a first way. Another is possible, along the same line, using the indices $n$ and $n^2$. Fat32 just did it, using prime delays. With more generality: take some signal $x[n]$, assuming that:


there exists  at least one non-square index $n_+$, for which $x[n_+] \neq 0$: for instance $x[-2]  =1$ (many negative numbers are not perfect squares),
$x$ is zero at all square indices:  $x[0] = x[1] = x[4] = x[9]=\cdots = 0$.


Then $x[n^2]$ is identically zero, while  $x[n]$ is not and the convolution is zero.