1. A method comprising: receiving target data comprising a plurality of observations, wherein each observation in said plurality of observation comprises (a) one or more values of one or more input neurons in a neural network and (b) one or more corresponding values of one or more output neurons in the neural network, and wherein the neural network comprises one or more layers of hidden neurons, one or more skip-layer connections between the one or more input neurons and the one or more output neurons, and one or more non-skip-layer connections between neurons of the neural network; determining an overall objective function that comprises (a) a first part corresponding to linear regression, (b) a second part corresponding to the neural network's unregularized objective function, and (c) a third part corresponding to a regularization of an unregularized objective function of the neural network; and determining, based at least in part on said target data and said overall objective function, an overall optimized first vector value of a first vector and an overall optimized second vector value of a second vector, wherein said first vector comprises a set of skip-layer weights for the one or more skip-layer connections and a set of output neuron biases, and wherein said second vector comprises a set of non-skip-layer weights for the one or more non-skip-layer connections; wherein the method is performed by one or more computing devices.