Is it okay to calculate the validation loss over batches instead of the whole validation set for speed purposes?
I assume you intended to write compute the evaluation metric over the validation set in batches; you do not compute loss over the validation set!
That is quite a standard practice in many academic implementations (because, when the validation set is large enough, the memory will be a constraint), however, be sure to take the average of the values over all the batches. Using a K-fold setup will increase the confidence in the reported values.