How to quantify phase distortion of a filter specific to a signal?
For a phase distortion metric I recommend using “group delay variation”. The definition of Group Delay is the negative derivative of phase with respect to frequency. The Group Delay is the delay in time that “group” of signals over a band of frequencies would have. A frequency response that is linear in phase (constant group delay with no variation) is not considered a distortion since all the frequency components would have the same delay so no actual distortion of the signal results. When the phase is not linear versus frequency (as in your plots) different frequency components of the signal arrive at different times at the output of the system, which can result in considerable distortion. Group Delay variation can be quantified as peak variation, peak to peak, or rms as in any other distortion metric.

As MattL points out in the comments below, a more comprehensive metric would be deviation from linear phase, where linear phase strictly means proportional to frequency with no phase offset term. If a phase offset exists, it would not appear in the result for the group delay computation yet indeed contribute to a distortion due to a varying delay versus frequency (the Hilbert Transformer is an excellent example of this: in order for all frequency components to have a 90° relationship with the input waveform, each component must have a different delay). For further details on this see Matt's answer here:

Group Delay for Hilbert Transformer and Resulting Dispersion