Adding noise to the input of a model trained with a regularized
  objective
  Regularization is a well studied problem in the context of neural networks.
It is usually used to improve the generalization performance when the number of
input samples is relatively small or heavily contaminated with noise. The
regularization of a parametric model can be achieved in different manners some
of which are early stopping (Morgan and Bourlard, 1990), weight decay, output
smoothing that are used to avoid overfitting during the training of the
considered model. From a Bayesian point of view, many regularization techniques
correspond to imposing certain prior distributions on model parameters (Krogh
and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective
function when a restricted type of noise is added to the input of a parametric
function, we derive the higher order terms of the Taylor expansion and analyze
the coefficients of the regularization terms induced by the noisy input. In
particular we study the effect of penalizing the Hessian of the mapping
function with respect to the input in terms of generalization performance. We
also show how we can control independently this coefficient by explicitly
penalizing the Jacobian of the mapping function on corrupted inputs.
