1. A system for training a machine learning model having parameters by determining a respective trained parameter value for each of the parameters of the machine learning model, the system comprising: a plurality of computing units; and a plurality of parameter server shards, wherein each shard executes on a respective computing unit, wherein each shard is configured to maintain values of a respective disjoint partition of the parameters of the machine learning model, and wherein a first shard of the plurality of shards is configured to perform operations comprising: receiving a first request for parameter values from a first replica of a plurality of replicas of the machine learning model, wherein each of the plurality of replicas implements an identical instance of the machine learning model with possibly different parameter values for the parameters of the machine learning model, and wherein the first request for parameter values is a request for parameter values for use in a particular iteration of a plurality of iterations of a machine learning training process to be performed by each of the replicas; in response to the first request, downloading a respective current value of each requested parameter to the first replica for use in performing the particular iteration of the machine learning training process; updating the current values of the parameters in the partition maintained by the parameter server shard based on one or more uploads of delta values computed by one or more replicas in the plurality of replicas to generate updated parameter values; receiving a second request for parameter values from a second replica of the plurality of replicas, wherein the second request for parameter values is also a request for parameter values for use in the particular iteration of the machine learning training process; and in response to the second request, downloading a respective updated value of each requested parameter to the second replica for use in performing the particular iteration of the machine learning training process.