Could I just choose the other (non-predicted) class when the accuracy is low?
The short answer is no, you shouldn't do that.
There is a &quot;distribution shift&quot; thing when you have different x-y relation on the validation set then on the train set. The distribution shift would deteriorate your model performance and you should try to avoid that. The reason it's bad - ok, you find the way to fix the model for validation data, but what about novel test data? Will it be like a train? Will it be like validation? You don't know and your model is worthless in fact.
What you can do

You should redefine the train/validation scheme. I would recommend making cross-validation with at least three splits
As mentioned in a comment, your model seems to overfit; try to apply some regularization techniques, like dropout, batch norm, data augmentation, model simplification and so on
Add some data if it's possible, that would better cover all the possible cases.