title: Tensorflow Mirror Strategy and Horovod Distribution Strategy
body: <p>I am trying to understand what are the basic difference between Tensorflow Mirror Strategy and Horovod Distribution Strategy.</p>
  
  <p>From the documentation and the source code investigation I found that Horovod (<a href="https://github.com/horovod/horovod" rel="noreferrer">https://github.com/horovod/horovod</a>) is using Message Passing Protocol (MPI) to communicate between multiple nodes. Specifically it uses all_reduce, all_gather of MPI. </p>
  
  <p>From my observation (I may be wrong) Mirror Strategy is also using all_reduce algorithm (<a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute" rel="noreferrer">https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute</a>).  </p>
  
  <p>Both of them are using data-parallel, synchronous training approach. 
  So I am a bit confused how they are different? Is the difference only in implementation or there are other (theoretical) difference? </p>
  
  <p>And how is the performance of mirror strategy compared to horovod?  </p>