Why is the gain of this LC two-port network infinite?
The gain is theoretically infinite, but even in theory it would take an infinite time to see the output reach an infinite amplitude.

In physical practice, there will be always be losses, which will limit the gain, even with a super-conducting L and a high quality vacuum capacitor.

Even in the absence of losses, each of the components will have a maximum limit.

As the energy in the circuit builds up, the current increases in the circuit. The power supply will need to support that current, no power supply can deliver an infinite load current. The wires connecting the components will melt at some finite current.

If the inductor is made on a core of iron or ferrite, then as the current builds, it will saturate, changing the L and detuning the resonance.

If you flip the L and C around, you have the business end of a 'magnifier' Tesla coil. When the energy in the capacitor, that is the voltage on it, gets high enough, pretty sparks break out from it. Any capacitor will have a maximum withstand voltage. No capacitor, unless it is has an infinite distance between its plates (that is, no real capacitor), can stand an infinite voltage.