Can the learning rate be considered both a parameter AND a hyper-parameter?
My understanding is that $\eta$ is set before the training starts to a large value but then, as the training progresses and the function gets closer and closer to a local minimum, the learning rate is decreased. In that case, doesn't the learning parameter satisfy both the definitions of a parameter and of a hyper-parameter?


No it does not, because you are at all times controlling the learning rate without reference to the data. Typically both the learning rate and the learning rate schedule are considered hyper-parameters, so by adding a learning rate schedule you have not transformed learning rate into a parameter, but instead added a new hyper-parameter that may need tuning!

The learning rate is also is not part of the model as it would be used in production, so it is a training hyper-parameter. Although that is not a strict separation, this also applies to things that are neither hyper-parameters nor parameters, such as the current momentum values, which are used to improve training rate, and are usually stored in a some data structures alongside the model being trained. Likewise most adaptive learning rate structures (used in e.g. RMSProp, Adagrad etc) are in this category - neither model parameter nor hyper-parameter.

It might be possible to argue that for a continuous online model, the learning rate could be a model parameter, as it is part of the production model being used to make predictions - provided it was still being used, but adaptive, depending on data seen to date, and/or current learning progress. Then if you had to feed in an initial learning rate you could maybe say that the learning rate was both a parameter and hyper-parameter of the ANN model. Although you would likely have to explain the scenario in full to be understood, it would be more usual to just say you were using an adaptive learning rate.