State of Stress in rotated 2D system
You are correct in your intuition regarding the signs of the forces.  The reason for two terms is due to the direction of the force.  The y-direction is not aligned with the face AB (it is aligned with y') so the force on this face is a combination of the forces normal to AB ($-\sigma_{x'x'}AB$) and the force along the face ($-\sigma_{x'y'}AB$).  Because the question wants only the force in the y-direction, we need to find the projection of these forces along that direction. 

If the angle between the coordinate axes is $\theta$, then the contribution of $-\sigma_{x'x'}AB$ in the y-direction is $-\sigma_{x'x'}AB\sin\theta$ and the contribution of $-\sigma_{x'y'}AB$ in the y-direction is $-\sigma_{x'y'}AB\cos\theta$, thus the total force in the y-direction is the sum of these contributions:

$$-\sigma_{x'x'}AB\sin\theta-\sigma_{x'y'}AB\cos\theta$$

Here is a quick figure to help visualize the geometry: