How do I properly tune parameters for a neural network?
It is totally hit and trail method. You have to play around it. There is no particular method to do that. Try to use GPU instead to CPU to compute fast such as "Google Colab". My suggestion is note down all the parameters that can be tunned.
eg:


Optimizer: Try to use different optimizer such as Adam,SGD,many more
learning rate : This is a very crucial parameter, try to change it from .0001 to 0.001 step by 0.0001.
Number of hidden layers : Try to increase no. of hidden layers.
Try to use Batch Normalization or Drop out or both if required.
Use correct Loss funtion.
Change Batch size and Epoch.
How do I tune the parameters for a neural network such as the amount of layers, the types of layers, the width, etc.?  Right now I simply guess for good parameters.  This becomes very expensive and time consuming for me as I will tune a network and then find out that it didn't do any better than the previous model.  Is there better way to tune the model to get a good test and validation score?