Difference amplifier configuration
This has got nothing to do with the op-amp being configured as a differential amplifier. Even in a simple non-inverting gain amplifier, as you increase the resistors (without changing the gain), the parasitic capacitors (input to ground and the parasitic feedback from output to inverting input) will cause the anticipated closed-loop gain to reduce. If you make the resistors smaller in value and push it too far, the op-amp won't be able to supply the current required by the negative feedback components.
As with most op-amp circuits, there is a goldilocks range of acceptable resistor values and, beyond that range, you start to get performance degradation.
Of course there are other things to consider; if you make the resistors too-high in value, input bias currents will start to degrade the DC accuracy of the circuit and this, in many cases is something to avoid.

Is there a deeper reason why setting all resistors to 100kΩ yields a
better frequency response?

Well, the best response you got was with 10 kΩ resistors: -

Again, it's a trade-off situation; there is a range of values that suit a particular application and most folk would default to 10 kΩ for the LM324. If it were a high-speed op-amp, you'd be looking at 1 kΩ resistors in the feedback (or even lower up at 1 GHz).