1. In a first processing node of a distributed data processing system having multiple processing nodes executing threads of a parallel job, a method for efficient inter-node communication among threads of the parallel job, the method comprising: generating, at the first processing node, one or more messages for transmission to at least one other processing node accessible via a network, wherein the one or more messages are generated by/for a first thread executing at the first processing node and the at least one other processing node executes one or more second threads of a same parallel job as the first thread; placing the one or more messages within a particular location in memory of the first processing node, wherein the placing of the one or more messages within the particular location triggers a transmission of the one or more messages; and in response to placing the one or more messages within the particular location in memory of the first processing node, transmitting, via a host fabric interface (HFI) of the first processing node, the one or more messages within a one-to-all messaging (OTAM) instruction broadcast on the network, wherein the one or more messages are transmitted to an HFI of one or more processing nodes in a cluster of processing nodes across the network that execute threads of the same parallel job as the first thread, wherein each of the cluster of processing nodes has allocated therein a plurality of receiving blocks to accept OTAM instruction broadcasts from other processing nodes in the cluster; wherein an OTAM instruction includes at least: a task/thread identifier (TID); an effective address corresponding to a real address at which a message is generated/stored; an effective address corresponding to a real address of the memory location at which one or more node addresses of one or more selected tasks of the one or more messages are located, an acknowledgement request indicator that indicates whether an acknowledgement is required to be received from each of the one or more processing nodes in the cluster; and one or more of: an OTAM message and status bits related to OTAM message, wherein the status bits include one or more of: a size of the OTAM message, a cache state information for data associated with the OTAM message, and a requirement for a synchronization operation following transmission of the OTAM message.