Advantages of square over circular engine nozzle?
Circular nozzles have these advantages:


They can be swiveled in any direction, so both pitch and yaw moments can be produced. Square nozzles only allow to create pitch forces, because their panels can only be moved around one axis. When moved differently in two-engined aircraft, limited roll moments are also possible.
Round nozzles are lighter. As long as exhaust gas pressure is higher than outside pressure, a square nozzle will be subject to bending loads across the width of its panels. This will make a square nozzle much heavier than a round nozzle.


Actually, square thrust vectoring nozzles have just one advantage: They can be made to reflect radar waves in a few well-defined directions, while round nozzles will scatter them all around. This makes square nozzles the preferred choice for stealth aircraft, regardless of thrust vectoring.


Engine nozzles of the F-22 (picture source). Note the serrated rear edges of the thrust vectoring panels.


Engine exhausts of the B-2. Since the B-2 uses four non-afterburing engines, fixed nozzles are used. Their location helps to protect the airplane from detection by infrared (IR) sensors, but makes thrust vectoring impracticable.


Engine nozzles of the YF-23 (picture source). The lower part is longer to shield the hot exhaust against IR sensors from below.

If the goal is to mostly reduce radar reflections in the forward hemisphere of the aircraft, round nozzles can be tolerated. Only when the aircraft needs to be protected from radar-guided missiles fired from behind will the square nozzles give a clear advantage. It seems that the people behind the Sukhoi PAK FA (and other stealth fighters like the Chinese J-20) do not worry so much about this detail.


Sukhoi T-50 (prototype for the PAK FA program) from below (picture by Dmitry Pichugin)


Chengdu J-20 rear view (picture source)

However, even Lockheed-Martin and Pratt &amp; Whitney now place maneuverability and weight over stealth in their nozzle design for the F-135 engine of the F-35 strike aircraft. Note the serrated edges on all contour steps. This helps to reduce radar returns, but is much less effective than a square nozzle. The benefit is less weight and better maneuverability.


F-35 with the F-135 nozzle visible (Picture source).