Speed decay proof for L2 regularization and non-normalizied weight initiation
Ok, I have found the source of confusion, thanks to @Emre for the insight.
The task talks about epochs of training, while I was trying to solve one batch update. I think this confusion is statistically possible enough to validate existence of this question :)