Filter module won't let me add nested values to my configuration
I figured out how to declare the filter settings schema by comparing with modules/filter/config/schema/filter.schema.yml and using the Schema docs.

Apparently my schema file simply needs to declare

filter_settings.xbbcode:
  type: filter
  label: 'XBBCode filter'
  mapping:
    [...]