I am looking to train a large face identification network. Resnet or VGG-16/19. TensorFlow 1.14

My question is - if I run out of GPU memory - **is it valid strategy to train sets of layers one by one?**

For example train 2 cnn and maxpooling layer as one set, then "freeze the weights" somehow and train next set etc..

*I know I can train on multi-gpu in tensorflow but what if I want to stick to just one GPU..*