Lowering learning rate makes my accuracy on the validation set go down
Optimizer functions in ML algorithms update themselves with learning rate to converge local minimums. When the step size (here learning rate = eta) gets smaller the function may not converge since there are not enough steps with this small learning rate (step size).

You should increase your learning rate or number of steps while keeping the learning rate constant to deal with the problem.

You may want to look at gradient descent: https://en.wikipedia.org/wiki/Gradient_descent

Also following image explains very well how to tune learning rate. (source: https://www.jeremyjordan.me/nn-learning-rate/)