What kind of tricks we can play with to further refine the trained neural network model so that it has lower objective function value?
I ask this question because many deep learning frameworks, such as Caffe, supports model refining function. For example, in Caffe, we can use snapshot to initialling the neural network parameters and then continue performing training as the following command shows:

./caffe train -solver solver_file.prototxt -snapshot snap_file.solverstate


In order to further train the model, the following tricks I can play with:


use smaller learning rate
change optimisation method. For example, change stochastic gradient descent to ADAM algorithm


Any other tricks I can play with? 

ps: I understand that reducing the loss function value of the training samples does not mean that we can get a better model. 

The question is way too broad, I think. However, this is a common practice, especially in case of a small training set. I would rank possible methods like this:


smaller learning rate
more/different data augmentation
add noise to train set (related to data augmentation, indeed)
fine-tune on subset of the training set.


The very last one is indeed a very powerful method to finalize the model that performs poor on some corner cases. You can then make a 'difficult' train subset in order to bias model towards it. I personally use it very often. 
