1. A method, implemented on at least one machine each of which has at least one processor, storage, and a communication platform connected to a network for distributed machine learning on a cluster including a plurality of nodes, the method comprising: determining a plurality of operation nodes from the plurality of nodes in the cluster, wherein each of the plurality of operation nodes is selected from a plurality of competing nodes of the cluster based on performance of each of the plurality of competing nodes on an identical subset of training data associated with a machine learning job; forming a network by connecting the plurality of operation nodes; distributing the machine learning job to the network, where each of the plurality of operation nodes performs a machine learning sub job on a respective subset of the training data; computing, in each of the plurality of operation nodes, a local parameter related to the machine learning sub-job; and generating an aggregated parameter related to the machine learning job based on the computed local parameters.