1. A system for training a model having parameters by determining a respective parameter value for each of the parameters of the model, the system comprising: a plurality of computing units; and a plurality of parameter server shards, wherein each shard executes on a respective computing unit, wherein each shard is configured to maintain values of a respective disjoint partition of the parameters of the model, wherein each shard is configured to perform repeatedly the following operations asynchronously with respect to every other shard: receive a succession of respective requests for parameter values from each of a plurality of replicas of the model, wherein each of the plurality of replicas is an identical instance of the model with possibly different parameter values for the parameters of the model; in response to each request, download a current value of each requested parameter to the replica from which the request was received; receive, from each of the plurality of replicas, a succession of uploads, each upload including respective delta values for each of the parameters in the partition maintained by the shard; and update values of the parameters in the partition maintained by the parameter server shard repeatedly based on the uploads of delta values to generate current parameter values.