Which NN would you choose to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$?
It depends on the complexity of your problem. $\mathbb{R}^2 \rightarrow \mathbb{R}^1$ looks simple, but I can give you some nonsense complicated examples that need a deep network. So, the complexity of the problem sets the number of layers and neurons. The kind of problem will determine the architecture of your network (if it needs memory or not). In most cases, the mean squared error is ok. However, for the activation function, I would go with the ReLU.
If the SVM is good enough for your problem, then go with it. Your question is general and an exact answer needs more information about the problem.