How do I swap Ctrl and Alt in just the Terminal?
iTerm2 will let you remap your modifier keys. This has the effect of just remapping them for terminals. Bonus: it's also a better terminal program for OS X than the built-in Terminal.app that ships with OS X.

You can alter the modifier keys on a per-profile basis as well. That lets you switch profiles for terminals depending on what you're doing with the terminal so you get the correct behaviour for local and remote sessions.