How do I declare my custom entities as non-multilingual?
Translatability for config and content entities is different. Translation handler is a content entity thing. Storage for translations needs to be explicitly enabled by adding a data table.

For config entities, it's different. Translatability works automatically there through config overrides. There's nothing that you need to do to opt-in to being translatable, but you also can't prevent it. What you can is control which properties are translatable through the config schema. Like the label,which is, technically, a translatable, user-provided string.

If you don't need it translated, just don't create any translations for it  or if you don't need any configuration translations, don't enable the config translation module.