Initialize perceptron weights with zero I'm new to datascience so please just don't blast me.

In a text book i found:


  Now, the reason we don't initialize the weights to zero is that the
  learning rate (eta) only has an effect on the classification outcome
  if the weights are initialized to non-zero values. If all the weights
  are initialized to zero, the learning rate parameter eta affects only
  the scale of the weight vector, not the direction.


Now, why? Are you able to explain me?
If you choose zero initial weights, then the perceptron algorithm's learning rate $\eta$ has no influence on a neuron's predicted class label.

This is because the decision function used in the perceptron algorithm depends only on the sign of $z$: 
$$
\phi(z) = \begin{cases} 1 &amp;\text{ if } z \ge 0\\ -1 &amp;\text{ otherwise.} \end{cases}
$$

Consider a perceptron algorithm with a single neuron.
(I will write ${\bf x} \cdot {\bf y}$ for the vector product ${\bf x}^T {\bf y}$ to avoid ugly double superscripts).


Pick an initial weight vector ${\bf w}^{(0)}.$
Plug in the first input vector ${\bf x^{(1)}}$ and predict the class label
$$
\hat{y}^{(1)}  = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}).
$$
This gives the weight update
$$
\Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)},
$$
where $\eta \in (0, 1)$ is the learning rate and $y^{(1)}$ is the true class label.
The new weights are ${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$ 
Similarly, plug in our the input vector ${{\bf x}^{(2)}}$ and predict the class label 
$$
\hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}).
$$ 
This gives the weight update
$$
\Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)},
$$
where $y^{(2)}$ is the true class label.


Notice that $\Delta{\bf w}^{(2)}$ implicitly depends on $\Delta{\bf w}^{(1)}$, which in turn implicitly depends on ${\bf w}^{(0)}$.
Let us unravel these dependencies by plugging in:
\begin{align*}
\Delta {\bf w}^{(2)} &amp;= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\
&amp;= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\
&amp;= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\
&amp;=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\
&amp;=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
\end{align*}

Now suppose we have initialized with zero weights: ${\bf w}^{(0)} = {\bf 0}.$
Since ${\bf 0} \cdot {\bf x}^{(1)} = 0$ and $\phi(0) = 1,$ the last line in this calculation simplifies to
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}
$$

Let us zoom in here and look at $\phi$ and its argument:
$$
\phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big).
$$
Since $\eta &gt; 0$, it does not change the sign of $(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$ But the sign is all that matters to the function $\phi$.
So we can simply remove $\eta$ from the function argument without changing the result:
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
$$

The same arguments hold for $\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$

It follows that
$$\Delta {\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$
for every $i = 1, 2, \dots$
This is what your quote means by saying that the parameter $\eta$ "affects only the scale of the weight vector, not the direction."

For nonzero initial weights, this is different. Recall the last line of the long calculation $\Delta {\bf w}^{(2)} = \dots$ above and zoom in again on the function $\phi$ and its argument:
$$
\phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big).
$$
If ${\bf w}^{(0)}$ is nonzero, then $\eta$ may affect the sign of the function argument, and therefore the predicted class label.