How to use a NN architecture that is too big for GPU?
I suggest a method similar to what @ignatius offered.
Since you don't need to train the first model and only the second one you could do the following:

Use the first model over the entire dataset and save the extracted results in your memory.
Train the second model using the results from the previous step.

This way you will only have to load one model at a given time.
In addition this will make the training of the second model to be faster.
Edit 1
(the op has specified he does not have enough memory for the inference results)
In this case I can suggest to parallelize between the inference of the first model and the training of the second model.
This could be achieved through prefatching.
Let's denote the original dataset as data1 and the corresponding inference results from the first model as data2.
Let's also denote the first model as M1 and the second model as M2.
Thus you can do the following:

load M1 to cpu, and M2 to gpu.
Draw a batch from data1 and use M1 to prepare a batch of data2 for M2.
Use the gpu to train M2 on the batch received from the previous step
while step 3 is being executed, prepare the next batch of data2 by executing step 2.

In this way, while M2 is being trained on a batch, the next batch is being prepared. This is called prefetching the next batch.
This will parallelize the training of M2 and the making of data2.
The method i described above has a very simple way to implement using the tensorflow &quot;Data&quot; api.
Here is a code example:
import numpy as np
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.data import Dataset

with K.device('cpu0'):
    input = Input((10,))
    x = Dense(100)(input)
    m1 = Model(input_shape=input, outputs=x)

with K.device('gpu0'):
    input = Input((100,))
    x = Dense(5)(input)
    m2 = Model(input_shape=input, outputs=x)


features = np.random.rand(10, 2000)
labels = np.random.randint(2, size=2000)
dataset = Dataset.from_tensor_slices((features, labels))


def preprocess_sample(features, label):
    inference = m2.predict(features, batch_size=1)
    return inference, label


dataset = dataset.shuffle(2000).repeat().map(preprocess_sample).batch(32)
dataset = dataset.prefetch(1)

m2.fit(dataset, epochs=100, steps_per_epoch=2000//32)