Unstable accuracy of CNN - When should I stop training?
If accuracy regresses something is wrong in either the network, or (more likely here) the meta-parameters (probably learning rate.) 

It can be difficult to tell when a model converges. I'd recommend looking at diagnostic graphs (typically training loss, training/validation accuracy, and ratio of weights:updates) over epochs. Typically convergence is considered when loss and accuracy level out and show diminishing returns beyond some threshold (your tolerance for 1.0e-x% improvements.) So, stop training/validation when it's improving less than what you care about.