File size: 989 Bytes
3d4d40d 29c9647 cf49f13 3d4d40d 29c9647 2a9343e 3d4d40d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Pipeline Parallellism with Controllable Memory
Pipeline Parallelism with Controllable Memory creates a framework on designing pipeline schedules and uses the framework to find memory optimal efficient schedules.
From our findings, we need approximately 1/3 memory under ideal conditions (F, B and W have same runtime), and 1/2 memory to create zero bubble schedule in realistic scenarios (with the necessary condition being W + 2B ≥ 2F and W + 2F ≥ 2B ).
Check out our paper at [Arxiv](https://arxiv.org/abs/2405.15362).
| Comparison assuming T_F=T_B=T_W | 1F1B | V-Min | V-Half | V-ZB |
| ----------------------------------------------------- | ------- |------- | ---------- | ---- |
| Bubble Rate | (p-1)/m | ???? | ??? | 0 |
| Activation Memory <br> (Compared to 1F1B) | p | (p+4)//3 | (p+2)//2 | p |
Bubble Rate here is calculated as (1 - longest stage time/(F+B+W)/m). |