How to reduce the size of Bert model(checkpoint/model_state.bin) using pytorch
I used torch.quantization.quantize_dynamic to reduce the model size but it is reducing my prediction Accuracy score.
I'm using that model file inside the Flask and doing some real time predictions, Due to the large size i'm facing issues while predicting. So could anyone please help me on reducing the bert model size using pytorch and guide me on who to do the real time predictions.

Here is the doc of torch.quantization.quantize_dynamic where dtype is set to torch.qint8.
so if you dont want your accuracy to drastically decrease use the below syntax
torch.quantization.quantize_dynamic(model, qconfig_spec=None, dtype=torch.float16)
