Performance metrics changing significantly based on batch size
Batch size is very related to the learning rate, especially in non-adaptive optimizers like the vanilla SGD that you are using.
I would suggest two alternatives:

Tune (reduce) the learning rate. You can check the answers to this SO question for some heuristics on how to choose the value. Apart from that, I recommend the article Don't Decay the Learning Rate, Increase the Batch Size.

Use an adaptive optimizer, like Adam.