Custom Display Modes on a custom entity type
As always, I found the answer just after posting the question.

For others looking for the same thing, you need to create an install .yml file in your custom module.
I found this by looking at the Media module in core.

In your module create config/install/core.entity_view_mode.ENTITY_TYPE_ID.NAME_OF_VIEW_MODE_TO_CREATE.yml that implements the following:

langcode: en # Or your default site language
status: false
dependencies:
  module:
    - my_module
id: entity_type_id.name_of_view_mode_to_create
label: 'My new View Mode'
targetEntityType: entity_type_id
cache: true


I hope this helps someone else too :D