Reducing the GPU memory usage when the model is already small enough
I cannot answer regarding the gap. What I can help you is how to reduce the memory usage. 

One thing you can try is you can convert your weights to fp16 and do inference with fp16. Don't worry you would not lose significant numerical precision especially since you are using that for deployment. In fact, this is how people do forward pass on mixed precision training. Theoretically this could increase speed and reduce memory usage. Hope it helps.