Learning rate of perception
After some study, I figured out the answer and want to share with people if someone also finds it helpful. The loss function of Perceptron is hinge loss or 

$J(w) = max(0, -yw^Tx)$. 

Adding a constant to the loss function does not change the function value, as it does not change the sign of the decision. In other words

$J_2(w) = max(0, -\alpha yw^Tx) = J(w)$.

If we do gradient descent using $J_2$, we have 

$\partial(J_2)/\partial(w) = 0$,  if $J_2 = 0$; 

$\partial(J_2)/\partial(w) = -\alpha yx$,  otherwise. 

So the update function of gradient descent is 

$w_{new} = w_{old} \pm \alpha x$. 

As long as $\alpha &gt; 0$, it does not change Perceptron decision in any step. This is why for Perceptron, you only need to set learning rate to be 1. 

Specifically answer the question, when people say "the learning rate only scales $w$", they are referring to $J_2(w) = max(0, -\alpha yw^Tx)$ rather than $w_{new} = w_{old} \pm \alpha x$.

A related question I found very helpful is  Normalizing the final weights vector in the upper bound on the Perceptron&#39;s convergence