


(a) Model graph



(b) Original execution order



(c)Optimized execution order
Figure 5: Overlapping Computation with Communication
3.4 Summary
Table 1 summarizes the optimizations discussed in this section and the corresponding performance bottlenecks addressed.


Optimization

Performance Bottleneck Addressed



Combining Input Sparse Features
   
Host-to-device memory copy
   


Horizontal fusion
   
GPU kernel launch overhead
   


Overlapping Computation with Communication
   
Embedding all-to-all access time
   

