Detection model - Training with class-instance limit-awareness
I can think of the following approach:

Let's say that you have two classes, A and B. Additionally, you now that for class A there is at maximum 5 instances (so 0, 1, 2, 3, 4 or 5) and for B 1 instance (0 or 1).

For this purpose, you can have 6 outputs for class A and 2 outputs for class B.
Between those 6 outputs for class A, only one should be active; same for B - only one of two should be active.

For example, if on some image there are 3 objects of class A and 0 for B, the outputs would be: [0, 0, 0, 1, 0, 0] for A, and [1, 0] for B (or something very close to 0s and 1s, right?)

With these outputs, you can also combine other outputs which are needed for detection.