Choosing an optimizer to perfectly fit a neural networks to training data
There's no free lunch, only bunch of tips. Your search space always depends on the problem you're tackling, and there's no one that fits them all. You need to run experiments to check this. Additionally, your net's architecture is one of the parameters, so it's hard to tell if your net will even be of sufficient capacity, without knowing the exact functions, you want to approximate. At first glance, optimizer is last thing you should worry about.