trying to decrease overfitting with regularisation in CNN I am doing transfer learning by retraining the publicly available inception layer, without regularisation here are my initial parameters and results:

training steps: 20,000
learning rate: 0.075
test accuracy :72.9%
train accuracy for final iteration : 98.0%
val accuracy for final iteration : 75.0%


clearly overfitting, so I tried L2 regularization. Here are my parameters and results, for the highest accuracy until now :

training steps: 40,000
learning rate: 0.1
test accuracy :71.1%
alpha for L2 : 0.00075
train accuracy for final iteration : 91.0%
val accuracy for final iteration : 71.0%


Looked promising so I decided to go further with the following hoping to get better accuracy :

training steps: 80,000
learning rate: 0.075
alpha for L2 : 0.005


go the following graph :



Since, the graph was getting saturated so I quit at 50,000 steps.

My question : 

Should I continue with it, hoping it will get better or should I try with other values(Please suggest some good range of parameters that I should try)? or may be other techniques like L1 regularization or dropout ?

Note : Since, it is real-world data so getting new data would be difficult, and I cant flip the image to generate new data since orientation is also a factor to determine the label for output classes
Steps for overcoming overfitting are the followings:  

 1. Add more data  
 2. Use data augmentation  
 3. Use architectures that generalize well  
 4. Add regularization  
 5. Reduce architecture complexity.


I would try to add Dropout, lately it is being the most used regularization technique in complex networks architectures. Start adding it in the later layers, since it is best no to lose initial info. Something like 0.6 or like that, and if it is not enough add dropout in the previous layer, and so on.