How does graphics api like OpenGL determine which triangle is back face to cull?
It's actually fairly simple (and in essence 'equivalent' to what you said above). If you represent triangle ABC's projected XY coordinates as a matrix...

$$M_{ABC} =   \begin{bmatrix}
    X_A &amp; X_B &amp; X_C \\
    Y_A &amp; Y_B &amp; Y_C \\
    1 &amp; 1 &amp; 1 \\
    \end{bmatrix}
$$

...then the sign of the determinant e.g. $X_A (Y_B - Y_C)+X_B(Y_C-Ya)+X_C(Y_A-Y_B)$ will allow you to decide how to cull. 
Further if the determinant is zero, then the on-screen triangle has zero area and can clearly be removed.

Another reason to use this notation is that the adjoint of $M_{ABC}$  gives you equations for the edges.