Distributed DL model with Tensorflow
I invite you to look at the Horovod project on github.
It's the most efficient way to currently execute distributed training with tensorflow. They have tutorials and benchmarks resource available