the advantage of being asynchronous, which allows a large number of
transfers to occur simultaneously, each at their own speed, without
blocking each other. It will only open pipes between pairs of nodes
when needed, on demand, and when one node fails only its incident
pipes will be closed, while all other ones will keep working as
normal. In addition, it is able to support multiple different
transports (TCP, of course, but also shared memory, NVLink,
InfiniBand, ...) and can automatically detect their availability and
negotiate the best transport to use for each pipe.
The TensorPipe backend has been introduced in PyTorch v1.6 and is
being actively developed. At the moment, it only supports CPU tensors,
with GPU support coming soon. It comes with a TCP-based transport,
just like Gloo. It is also able to automatically chunk and multiplex
large tensors over multiple sockets and threads in order to achieve
very high bandwidths. The agent will be able to pick the best