Learning rate and memory of Least Mean Square algorithm
This should be the weight update formula your model uses.

$W_{n+1} = W_n - \mu\nabla\epsilon[n]$,

$\mu$ is the learning rate, and we multiply it with the change we want to do. Obviously, the smaller the learning rate you have, the less update you make, and thus the "longer" the model converges and also the "longer" the model remembers its previous states.

Think like this, if you set the learning rate to 0, you learn nothing and your parameters are never updated.