1. A method for providing distributed learning over a plurality of parallel machine network nodes, comprising: training the parallel machine network nodes in a peer-to-peer fashion and allocating a per-sender receive queue at every machine network node and performing distributed in-memory training; and training each unit replica and maintaining multiple copies of the unit replica being trained, wherein all unit replicas train, receive unit updates and merge in parallel in a peer-to-peer fashion, wherein each receiving machine network node merges updates at a later point in time without interruption and wherein the propagating and synchronizing unit replica updates are lockless and asynchronous.