1. A system for training a deep network having parameters by determining a respective parameter value for each of the parameters of the deep network, the system comprising: a plurality of identical model replicas, wherein each of the plurality of replicas is an identical instance of the deep network with possibly different parameter values for the parameters of the deep network, wherein each model replica executes on a respective computing unit, wherein each model replica is configured to operate independently of each other model replica, and wherein each model replica is further configured to perform repeatedly the following operations: receiving, from a parameter server, current values of one or more of the parameters of the deep network, wherein the parameter server is configured to maintain values of the parameters of the deep network; computing respective delta values for each of a plurality of the parameters of the deep network by performing one or more iterations of a training process; and providing, for each of the plurality of the parameters of the deep network, the delta value for the parameter to the parameter server.