Prior pipeline parallel systems use a fixed number of micro-batches per mini-batch ( ). GPipe suggests , where  is the number of partitions (pipeline length). However, given that PipeTransformer dynamically configures , we find it to be sub-optimal to maintain a static  during training. Moreover, when integrated with DDP, the value of  also has an impact on the efficiency of DDP gradient synchronizations. Since DDP must wait for the last micro-batch to finish its backward computation on a parameter before launching its gradient synchronization, finer micro-batches lead to a smaller overlap between computation and communication. Hence, instead of using a static value, PipeTransformer searches for optimal  on the fly in the hybrid of DDP environment by enumerating  values ranging from  to . For a specific training environment, the profiling needs only to be done once (see Algorithm 1 line 35).