ower, because submodules are moved to GPU as needed, and immediately returned to CPU when a new module runs.
Full-model offloading is an alternative that moves whole models to the GPU, instead of handling each model’s constituent modules. This results in a negligible impact on inference time (compared with moving the pipeline to cuda), while still providing some memory savings.
In this scenario, only one of the main components of the pipeline (