1. A method for providing distributed learning over a plurality of parallel machine network nodes, comprising: providing a vector object library with vector abstractions to create unit parameters or gradients to be synchronized across the parallel machine network nodes; training the parallel machine network nodes in a peer-to-peer fashion and allocating a per-sender receive queue at every machine network node and performing distributed in-memory training; training each unit replica and maintaining multiple copies of the unit replica being trained, wherein all unit replicas train, receive unit updates and merge in parallel in a peer-to-peer fashion, wherein each receiving machine network node merges updates at a later point in time without interruption; and propagating and synchronizing unit replica updates as lockless and asynchronous operations using wait-free queues, wherein multiple per-sender queues are combined with remote direct memory access (RDMA) circuit.