Can I modify a PC AT switching power supply to put out higher amps?
The transformer and the internal circuitry design of the PSU will not allow you to simply get &gt;8A from the 12V output.

So if my math is correct, 5V * 30A = 150W and 12V * 8A = 96W. If I can combine these into just 12V, I should get 20A and that's way more than I need, I only need about 12~15.

It's not that simple. After boosting the 5V to 12V, you can get 11A at best with a properly designed step-up converter. However, you shouldn't connect this 12V output to the main 12V output as they are not ideal sources. Plus, as they are being regulated by individual converters, there may be some other problems regarding stability.

Q. is there a way to modify this old AT PS so that it outputs about 15A on 12V? Maybe a boost converter, maybe there's a way to modify it internally, IDK.

Simply no. Removing all the secondaries of the main transformer and re-winding an individual secondary with thicker wires, replacing the secondary rectifier diodes with higher-current ones, adding parallel output capacitors (due to increased output current) then modifying the control circuitry (e.g. tricking POWER_GOOD signal) may be a solution. But this still requires knowledge of SMPS design and especially the PSU's internal circuitry.