How does graphics api like OpenGL determine which triangle is back face to cull? When I was writing a software rasterizer for my university class, we did culling in  eye coordinates. Essentially, we use cross product to get a normal from the 3 vertices (in eye coordinates), then we dot product that normal with one of the 3 vertices (since we are in eye coordinate, that vertex is essentially a vector from eye [0, 0, 0,.1] to triangle surface). If the dot product is > 0, meaning the normal and the eye vector point in the same direction, we cull that triangle.

OpenGL however doesn't do culling in eye space, since culling is after vertex shader. So all they have are vertices in clip space. So how do they know which triangle to do culling? Our triangles, when fed into the graphic pipeline, are always in CCW order. How do some turn into CW order?
It's actually fairly simple (and in essence 'equivalent' to what you said above). If you represent triangle ABC's projected XY coordinates as a matrix...

$$M_{ABC} =   \begin{bmatrix}
    X_A &amp; X_B &amp; X_C \\
    Y_A &amp; Y_B &amp; Y_C \\
    1 &amp; 1 &amp; 1 \\
    \end{bmatrix}
$$

...then the sign of the determinant e.g. $X_A (Y_B - Y_C)+X_B(Y_C-Ya)+X_C(Y_A-Y_B)$ will allow you to decide how to cull. 
Further if the determinant is zero, then the on-screen triangle has zero area and can clearly be removed.

Another reason to use this notation is that the adjoint of $M_{ABC}$  gives you equations for the edges.