Strange behavior with Adam optimizer when training for too long
This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.

One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the current gradient to decide the current step. However, when your gradient becomes and stays very close to zero, this will make the squares of the gradient become so low that they either have large rounding errors or are effectively zero, which can introduce instability (for instance a long-term stable gradient in one dimension makes a relatively small step from $10^{-10}$ to $10^{-5}$ due to changes in other params), and the step size will start to jump around, before settling again.

This actually makes Adam less stable and worse for your problem than more basic gradient descent, assuming you want to get as numerically close to zero loss as calculations allow for your problem.

In practice on deep learning problems, you don't get this close to convergence (and for some regularisation techniques such as early stopping, you don't want to anyway), so it is usually not a practical concern on the types of problem that Adam was designed for.

You can actually see this occurring for RMSProp in a comparison of different optimisers (RMSProp is the black line - watch the very last steps just as it reaches the target):



You can make Adam more stable and able to get closer to true convergence by reducing the learning rate. E.g. 

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)


It will take longer to optimise. Using lr=1e-5 you need to train for 20,000+ iterations before you see the instability and the instability is less dramatic, values hover around $10^{-7}$.