Can a Tensorflow saved model created on a large GPU be used to make predictions on a small CPU?
Resources needed for prediction depend on the model size - not on the training device.
If the model has 200 bln variables - you will not be able to run it on workstation (because you have not enough memory).
But you can use model with 10 mln variables with no problems even if it was trained on GPU or TPU.
Every variable takes 4 to 8 bytes. If you have 8 GB of memory - you will probably be able to run a model with hundreds million variables.
Prediction is fast (assuming you have enough memory). Resources needed to train model quickly. It is efficient to train on GPU/TPU even if your model is small.
Let’s say one saves a Tensorflow model created using large data and GPUs. If one wanted to then use the saved model to do a single prediction with one small piece of data, does one still need the huge computers that created the model?
I’m wondering more generally how the size and computing resources needed to generate a deep learning model relate to using the model to make predictions.
This is relevant because if one is using Google Cloud Compute it costs more money if one has to use the huge computers all the time. If one could just use the huge computers to train the model and then more modest ones to run their app that make predictions it would save a lot of money.