Confusion on Delta Rule and Error
You are right, the least square solution need not be unique as you have illustrated. 

In general, we are trying to solve for $w$ in the system $$Xw=t$$

which of course need not even be consistent (meaning has a solution). In the event that it has a solution, the uniqueness is also not guaranteed if the matrix $X$ has non-zero vector in the nullspace of $X$.

For the least square solution, we are trying to minimize

$$\frac12(Xw-t)^T(Xw-t)$$

of which the minimal is attained when $$X^T(Xw-t)=0$$

that is $$X^TXw=X^Tt$$

and it is only unique when $X^TX$ has full column rank. However, note that uniqueness is not needed for gradient descent to work.