Applying activation on part of the layer in Keras
YOLO architecture uses the softmax activation function determining the classes of objects in bounding boxes in the output layer. But from the code you have shared, sigmoid is used in the last layer for prediction. It seems that you're trying to implement YOLOv3.
Please look at this paper original paper. This will help to give a better understanding.
Authors of YOLOv3 have refrained from softmaxing the classes since the method rests on the assumption that classes are mutually exclusive. For example, if there are classes like “cat” and “animal” in the dataset and one of the objects in the bounding boxes is a cat, this assumption fails, because a cat is also an animal. Instead, independent logistic classifiers predict each class score and a threshold is used to perform multilabel classification for objects detected in images. An element belonging to a certain class will not be influenced by the decision of that element belonging to another class (binary cross-entropy loss).