State of Stress in rotated 2D system I'm working on the rotation of the state of stress using the book Geodynamics of Turcotte page 101. The 2D coordinate system $x'$ and $y'$ is inclined at an angle $\phi$ with respect to the $x$, $y$ coordinate system (see Figure 2.14).

The force in the $y$-direction on face $AO$ is

$$ \sigma_{yy}AO $$

and the force in the $y$-direction on face $OB$ is

$$ \sigma_{xy}OB $$

The force in the $y$-direction on face $AB$ is 

$$ -\sigma_{x'x'}AB\cdot\sin\theta -\sigma_{x'y'}AB\cdot\cos\theta$$

I don't understand why the force in the $y$-direction on face $AB$ is equal to the above equation? 

I think the negative signs are there because the stresses are not oriented in the same direction than the $y$-direction. I don't know why it is necessary to subtract the two lengths.
You are correct in your intuition regarding the signs of the forces.  The reason for two terms is due to the direction of the force.  The y-direction is not aligned with the face AB (it is aligned with y') so the force on this face is a combination of the forces normal to AB ($-\sigma_{x'x'}AB$) and the force along the face ($-\sigma_{x'y'}AB$).  Because the question wants only the force in the y-direction, we need to find the projection of these forces along that direction. 

If the angle between the coordinate axes is $\theta$, then the contribution of $-\sigma_{x'x'}AB$ in the y-direction is $-\sigma_{x'x'}AB\sin\theta$ and the contribution of $-\sigma_{x'y'}AB$ in the y-direction is $-\sigma_{x'y'}AB\cos\theta$, thus the total force in the y-direction is the sum of these contributions:

$$-\sigma_{x'x'}AB\sin\theta-\sigma_{x'y'}AB\cos\theta$$

Here is a quick figure to help visualize the geometry: