Non-linear Regression
To answer you first question about Non linear regression:

I believe your problem of choosing mapping function for non linear regression can be solved by using Support Vector Machines. 

SVMs can learn non linear mapping functions in a kernel-induced feature space. 
What this means is in svms , the basic idea is to map the input data X into some high dimensional feature space f using a non linear mapping (kernel) and then doing linear regression in this feature space. 

To learn more about non-linear regression and kernels, you can read this.

Secondly, Regularization is a technique that is used to solve over-fitting problem. This usually happens when you use a very dense model for your training set or you train the model for far too many steps. In this case, while the accuracy on your train set is high,but it performs very poorly in case of unseen data. Hence when you add regularization, it helps reduce the cost function.

Regularization is of two types, L1 and L2. The difference lies in the power of weight-coefficients.These should be enough for your SVM based models.

To reduce overfitting induced high cost, you can also use BatchNormalization and Dropout algorithms.

Hope this helps :)