Training infrastructure. Assume the training infrastructure contains a GPU cluster that has  GPU servers (i.e. nodes). Each node has  GPUs. Our cluster is homogeneous, meaning that each GPU and server have the same hardware configuration. Each GPU's memory capacity is . Servers are connected by a high bandwidth network interface such as InfiniBand interconnect.