How to use Adam optim considering of its adaptive learning rate?
In the Adam optimization algorithm, the learning speed is adjusted according to the number of iterations. I don't quite understand Adam's design, especially when using batch training. When using batch training, if there are 19,200 pictures, each time 64 pictures are trained, it is equivalent to 300 iterations. If our epoch has 200 times, there are a total of 60,000 iterations. I don't know if such multiple iterations will reduce the learning speed to a very small size. So when we are training, shall we initialize the optim after each epoch, or do nothing throughout the process?

Using pytorch. I have tried to initialize the optim after each epoch if I use batch train, and I do nothing when the number of data is small.

For expample, I don't know whether the two pieces of code is correct:

optimizer = optim.Adam(model.parameters(), lr=0.1)
for epoch in range(100):
    ###Some code
    optim.step()


Another piece of code:

for epoch in range(100):
    optimizer = optim.Adam(model.parameters(), lr=0.1)
    ###Some code
    optim.step()


You can read the official paper here https://arxiv.org/pdf/1412.6980.pdf

Your update looks somewhat like this (for brevity, sake I have omitted the warmup-phase):

new_theta = old_theta-learning_rate*momentum/(velocity+eps)


The intuition here is that if momentum>velocity, then the optimizer is in a plateau, so the the learning_rate is increased because momentum/velocity &gt; 1. on the other hand if momentum&lt;velocity, then the optimizer is in a steep slope or noisy region, so the learning_rate is decreased.

The learning_rate isn't necessarily decreased throughout the training, as you have mentioned in you question.
