Is there a possibility that older OS X apps will break in the new macOS?
Yes applications can break when a new OS is introduced. The APIs that it calls could have been changed or possibly removed.

This is why many apps will get updated just after an OS release and why developers get a beta test of the OS so they can check.

If the developer is not active anymore then you have to be the one who tests it. In large organisations there is often a complex plan to upgrade an OS: all the applications that the organisation uses have to be tested before agreement to update the OS is given. This is one reason many organisations have stayed on old versions of Windows (e.g. Windows XP) because they can't get all apps to pass or no-one has given the budget to test.