How does this impedance matching circuit work? How to simulate it correctly in SPICE?
Firstly let's look at the coax in your question: -



With lumped parameters of 36.8 nH and 31.2 pF, the actual characteristic impedance can be calculated by: -

\$Z_0 = \sqrt{\dfrac{L}{C}}\$ = 34.34 ohms.

So, clearly it is not intended to match the signal because there is a significant mismatch.

Next the amplifier (INA240A). The data sheet tells us that it doesn't provide any useful signal outputs above 1 MHz and for a transmission line to be "significant" at 1 MHz, it would be longer than about one-tenth of the wavelength of 1 MHz (300 metres) at 30 m or more.

So, unless you have a very long piece of coax there is nothing about the attenuator that is related to impedance matching.

Let's say your cable was 300 metres plus. Because we are talking about fairly low frequencies (less than 1 MHz), the coax's characteristic impedance would not look like 50 ohm or 75 ohm - it would have a complex impedance and this is also not-matched by the 75 ohm resistor.

In short, my conclusion is that R2 and R3 are just a simple attenuator and that anything told to you about why it is there is a red-herring.