How to increase model's test accuracy?
While Yolov3 is a great model it may underperform for certain datasets. A simple example would be this. Take a model and train it on cifar 10. The same model might significantly perform lower on cifar 100 or imagenet!
Things you could do to improve your model:

Introduce Batch or group normalization?
Study the data better, if possible see if the data is used elsewhere and how they approach it.
Epochs isn't the problem, so that should stay the same.

If nothing continues to work then you might have to opt for some other model. Try yolov5 or rcnn!