Why would Pytorch be non-deterministic for batch only?
The cudnn implementaton of the LSTM has determinism issues that appear to be fixed in the 7.6.1 release. Check your cudnn version.

https://github.com/pytorch/pytorch/issues/18110