Moving a vertex through the cursor
Figured it out, I did skip a step, you are supposed to multiply by the homogeneous coordinate.
Basically the actually projected point is (courtesy of GLM):
        vec&lt;4, T, Q&gt; tmp = vec&lt;4, T, Q&gt;(obj, static_cast&lt;T&gt;(1));
        tmp = model * tmp;
        tmp = proj * tmp;

        tmp /= tmp.w;

Kinda obvious in hindsight.