Should your learning rate get smaller as your neural net gets larger?
The learning rate affects how much your model weights are changed with each update. So the question is what is a good learning rate? This depends on your problem space, dataset and network parameters (ie. Activation functions, complexity, regularisation, optimizers, etc).
Too high a learning rate, your model may not reach convergence due to weights updated to “overshoot” the optimum.
Too low a learning rate, your model may also not reach convergence due to taking too many epochs, as the updates to weights are tiny.
It is best to experiment with the defaults ~0.01, and adjust accordingly based on visualizing the validation loss and training loss change over epochs. Also try out different optimizers (Adam, RMSprop, etc) as these adapt your learning rate over time.
Another approach is to implement a learning rate scheduler, that decays your learning rate over time - the idea being as a model nears the global minima, you want to fine tune its descent with smaller weight updates.