1. A method of training a Deep Neural Network (DNN) model, comprising: at a head computing device comprising one or more processors and memory: establishing an initial DNN model; dividing a training data corpus into a plurality of disjoint data subsets; for each of the plurality of disjoint data subsets, providing the data subset to a respective training processing unit of a plurality of training processing units operating in parallel, wherein a training processing unit is provided by a respective leaf computing device of a plurality of leaf computing devices operating in parallel, the plurality of leaf computing devices being coupled to the head computing device, and wherein the respective training processing unit applies a Stochastic Gradient Descent (SGD) process to update the initial DNN model to generate a respective DNN sub-model of a plurality of DNN sub-models based on the data subset; and merging the respective DNN sub-models generated by the plurality of training processing units to obtain an intermediate DNN model, wherein the intermediate DNN model is established as either a respective initial DNN model for a next training iteration or a final DNN model in accordance with a preset convergence condition, and wherein merging the respective DNN sub-models comprises: for each DNN sub-model of the plurality of DNN sub-models, assigning a respective first merging weight to each layer of a plurality of layers of each DNN sub-model; and for the plurality of DNN sub-models, applying a linear combination of the respective DNN sub-models to obtain the intermediate DNN model, wherein applying the linear combination comprises: assigning a respective second merging weight to each DNN sub-model of the plurality of DNN sub-models, wherein the respective second merging weight of each DNN sub-model is a vector of the respective first merging weights of all layers of the respective DNN sub-model, wherein the initial and final DNN models are acoustic models of speech recognition and the training data corpus comprises a plurality of randomized speech files.