Why is the frequency response different based on the amplitude of the impulse I feed into the filter?
The lower amplitude the input signal has compared to the resolution of the filter coefficients the weaker is its ability to shape the frequency of the input signal. It also depends on the span of the filter coefficients.

You get the same result if you feed the filter with 32-bit coefficients with a signal having an amplitude covering only the lower 16-bits. If you want to work around this effect you need to amplify the input signal.