1. A computer-implemented method of executing an application program on a parallel computing system having a plurality of compute nodes, each node having at least a processor and a memory interconnected by at least one data communications network, comprising: dividing a program binary into a plurality of executable code fragments; distributing each of the plurality of executable code fragments to at least one of the plurality of compute nodes, wherein the plurality of compute nodes are connected by communication links forming a binary tree; initiating execution of the application program on the processor on each of the plurality of compute nodes; receiving, by at least a first compute node and a second compute node, at least a first one of the executable code fragments; storing the first executable code fragment in a respective memory space of the first compute node and the second compute node; during execution of the application program determining, by a third compute node, a need for the first executable code fragment; selecting, by the third compute node, to request the first executable code fragment from either the first compute node or second compute node, based on the relative distance in the binary tree between the third compute node and the respective first and second compute nodes; receiving, by the selected compute node, a request from a third compute node for the executable code fragment; and transmitting, by the selected compute node, the requested code fragment to the third compute node.