Why do stealth aircraft generally lose in maneuverability?
Radar stealth is achieved by carefully designing the aircraft to reflect radar waves in specific directions. It helps to keep the aircraft shape as simple as possible.  

External weapons make this more complex: each missile adds a large number of surfaces that can reflect radar. The missile can also interact with the airplane fuselage to reflect radar waves several times. And once the missile has been launched it leaves an empty pylon behind with yet more surfaces capable of reflection.
It's much easier to maintain stealth if the missiles are stored inside the aircraft. 

Some aircraft (e.g. the F-117) also try to achieve IR stealth by hiding the exhaust, mixing the exhaust gases with air etc. This is difficult to combine with vectored thrust. 

Still, an aircraft like the F-22 has pretty good maneuverability, thanks to good aerodynamics and huge amounts of engine power. 

The Su-37 uses vectored thrust and canards (again, extra reflection surfaces) to achieve high maneuverability.

So, there's a tradeoff going on. Do you want to blow your enemy out of the sky before he's seen you (which requires stealth)? Or do you focus on close combat (where maneuverability is most important)?