Assign Keras/TF/PyTorch layer to hardware type
Suppose we have the following architecture:


Multiple CNN layers
RNN layer
(Time-distributed) Dense classification layer


We want to train this architecture now. Our fancy GPU is very fast at solving the CNN layers. Although using a lower clockrate, it can perform many convolutions in parallel, thus the speed. Our fancy CPU however is faster for the (very long) resulting time series, because the time steps cannot be parallelized, and the processing profits from the higher CPU clockrate. So the (supposedly) smart idea for execution would look like this:


Multiple CNN layers (run on GPU)
RNN layer (run on CPU)
(Time-distributed) Dense classification layer (run on GPU/CPU)


This lead me to two important questions:

Is it possible, with any of the frameworks mentioned in the title, to distribute certain layers to certain hardware, and how?

If it is possible, would the overhead for the additional memory operations, e.g. tranferring between GPU-/CPU-RAM, render the whole idea useless?

Basically, in Pytorch you can control the device on which variables/parameters reside. AFAIK, it is your responsibility to make sure that for each operation all the arguments reside on the same device: i.e., you cannot conv(x, y) where x is on GPU and y is on CPU.

This is done via pytorch's .to() method that moves a module/variable .to('cpu') or .to('cuda:0')
