Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions? I have trained two text classification models using GPU on Azure. The models are the following

Bert (ktrain)
Lstm Word2Vec (tensorflow)

Exaples of the code can be found here: nlp
I saved the models into files (.h5) for later use. The files are big e.g. 27,613kb for the lstm and 1.2 gb for bert.
I loaded the models and in a computer where only CPU is available. They both work fine but the model.predict(text) function is super slow predicting the class of the text e.g. on average 1 tweet sized message per second.
Adding GPU on the computer is not an option. I wonder if there is another way to make it run faster? e.g. train the models in a different way (without compromising accuracy) or save the model in a different file format?
There are a couple of options:

Optimize tensorflow for your specific CPU. Sometimes the official versions of tensorflow are not compiled with support for some instruction sets (e.g. SSE4.1, SSE4.2, AVX, AVX2, FMA). Usually, there is a tensorflow runtime warning message stating so. This prevents some computations to take place in parallel. You can either download a version that is optimized for your CPU (e.g. from intel) or you can compile tensorflow yourself.

Prune the model. BERT is a multi-head attention model, and it is possible to remove some of its attention heads while retaining most of the quality. Here's a blog post that shows how to do it for tensorflow.