Choosing an optimizer to perfectly fit a neural networks to training data In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae &lt; 1e-04) on the training data? 

Details: In my particular case, the input of the function is 60-dimensional and the output is 1-dimensional (the training data set is prepared by solving a forward model). The input and output are normalized and the sample mean is converted to 0. My neuron's activation function is tanh (this gave me a better result than ReLu and Sigmoid). I have so far used Keras's "adam" and "sgd" as optimizers and I have tried with various learning rates. 

In addition, I have tried with increasing the number of neurons in each hidden layer and increasing the number of hidden layers as well. At some point, the total number of my trainable parameters was more than 100 million. 

However, even for trying different batch sizes and 10,000 epochs, my best mean absolute error (mae) was never below 0.07. The only part I haven't yet experimented much is to customize/use advance optimizers. I am completely clueless why the neural network can't find a set of trainable parameters where, at least, it can over-fit the training data? 

Any suggestions from the experts? Thanks in advance for your time and patience. I really appreciate your support. 

An example code:

input_size = ND
nodes = 10000
inp = Input(shape=(input_size,),name='Input')
l0 = Dense(nodes, activation='tanh',name='Level0')(inp)
l1 = Dense(nodes, activation='tanh',name='Level1')(l0)

lo = Dense(1, activation='tanh',name='Level_out')(l1)

merged = Model(inputs=inp,outputs=l10)
opt = optimizers.adam(lr = 1e-5)
merged.compile(optimizer=opt,
              loss='mean_squared_error',
              metrics=['mae'])
There's no free lunch, only bunch of tips. Your search space always depends on the problem you're tackling, and there's no one that fits them all. You need to run experiments to check this. Additionally, your net's architecture is one of the parameters, so it's hard to tell if your net will even be of sufficient capacity, without knowing the exact functions, you want to approximate. At first glance, optimizer is last thing you should worry about.