1. A computing device comprising a recognition system, the recognition system comprising: a deep neural network (DNN) trained using training data corresponding to a plurality of users, the DNN parameterized by a plurality of parameters and comprising multiple layers, the multiple layers comprising: an input layer that receives features of a captured observation; a plurality of hidden layers, each hidden layer comprising a respective plurality of nodes, each node in a hidden layer being configured to perform a nonlinear transformation on output of at least one node from an adjacent layer in the DNN; and an output layer that comprises models of context-dependent units, the output layer receiving outputs of nodes from an uppermost hidden layer in the DNN, wherein output of the DNN comprises a probability distribution over the models of the context-dependent units; and an adapter component that conservatively adapts at least one parameter of the plurality of parameters to form an adapted DNN, wherein the adaptor component conservatively adapts the DNN such that deviation between outputs of the DNN and the adapted DNN is constrained.