STM32F412 CAN Mailbox Runs Full
You only have three transmit mailboxes.
That means that you can set only three frames pending for transmit by the bxCAN controller.  Others have to wait in software buffers.
If you did not implement the error handlers of CAN bus and cancel transmission when they fail (eg: bus full, no acknowledge or arbitation failure) you will eventually use up all mailboxes.
The hardware does not provide a timeout function, you have to keep track of possible mailbox timeouts yourself.
Your code only aborts transmit of box 0 when all mailboxes are full, so your behavior is not strange.
Also, if you did enabled the automatic retransmit feature, mailboxes will never free automatically until transmit is successfully completed.


Also careful, there are only two FIFO's of 3 object deep for receiving. Without fitlers only one is used. And this can be overrun these if you some have interrupt jitter and high bus speed.