How to determine if the system is linear or nonlinear
As Matt L. says you'll need to check for homogeneity and, possibly, additivity.

Homogeneity

That test says that if:
$$
y[n] = f(x[n])
$$
then
$$
A \cdot y[n] = f(A \cdot x[n])
$$
for all scalar $A$.

Additivity

This test says that if
$$
y_1[n] = f(x_1[n])
$$
and
$$
y_2[n] = f(x_2[n])
$$
then
$$
y_{\rm tot}[n] = f(x_1[n] + x_2[n]) = y_1[n] + y_2[n]
$$

You just need to apply these for your system.



I say "possibly" because if the system fails one of these tests, then it will not be linear --- so there is little point testing the second condition.