Output of a linear time-invariant(LTI) system
This question could be rephrased as is $y(t) = T{x(t)} = x(t) ^ 2$ LTI system. This is a textbook case of a non-linear system. Here is a quick proof:
For a system to be linear the following must be true:
$T{a \cdot x(t)} = a \cdot T{x(t)}$ . i.e. if you input scaled signal into linear system you should get the same output as if you inputted that same signal but unscaled, and then scaled the resulting output. This is referred to as the scaling property.
We can use this test on the system in your exam question $y(t) = T{x(t)} = x(t) ^ 2$ :
$y(t) = T{a \cdot x(t)} = (a \cdot x(t)) ^ 2 = a^2 \cdot (x(t))^2$
$y(t) = a \cdot T{x(t)}= a \cdot (x_1(t)) ^ 2 = a \cdot (x_1(t))^2$
As we can see the results are not equal. Therefore the system is not linear, and so it is not LTI.
It appears that your teacher may have wanted to approach this problem in a different manner, so I apologize if this solution isn't helpful. However, this approach is completely valid and much quicker in my opinion. After getting more comfortable with it you can identify operators such as a square and immediately recognize that the system is not an LTI.