Using comment view modes
There isn't a way to do this without writing your own formatters. However, could you open a feature request in the core issue queue as this is a worthy configuration option for the existing formatter.