How can I offset/shrink a triangular polygon in GLSL?
Given triangle ▲ABC, we bisect angle ∠BAC with line AD, derived with Angle Bisector Theorem:

BA / BD = CA / CD

Point E represents our objective refined position on the desired resulting inset triangle. As it lies upon angle bisector AD, it is equidistant from sides BA &amp; CA, forming identical right triangles ▲AFE &amp; ▲AGE. We can now use Sine for Right Triangles to find the length of AE:

AE = EG / Sin(∠EAG)

That's all the math we need, so let's cook up some GLSL!

We start off with all the typical attributes: position, normal, and transformation matrices, but since the vertex shader only works on a single vertex, we need to add the neighboring vertices as additional attributes. This way, each vertex will find its own "point E", creating the resulting inset triangle. (Note: I don't call them "B" &amp; "C" here, because they are not yet in screen space.)

    attribute vec3 left; //vertex to the left of this vertex
    attribute vec3 right; //vertex to the right of this vertex


Speaking of screen space, I'm also including the aspect ratio of the display, (and making it a uniform, in case the window is resized.)

After preparing varying normals for the fragment shader, and transforming the face into clipping-space, we can get down to the business of applying the above math:

        attribute vec3 left; //vertex to the left of this vertex
        attribute vec3 right; //vertex to the right of this vertex
        uniform float aspect;
        varying vec3 vNormal;
        varying vec2 vUv;

        void main() {
            vNormal = normal;
            vUv = uv;

            mat4 xform= projectionMatrix * modelViewMatrix;
            vec4 A = xform * vec4( position, 1.0 );
            vec4 B = xform * vec4( left, 1.0 );
            vec4 C = xform * vec4( right, 1.0 );

            vec3 CB = C.xyz - B.xyz;
            vec2 BA = B.xy - A.xy;
            vec2 CA = C.xy - A.xy;
            float lengthBA = length(BA);
            float lengthCA = length(CA);
            float ratio = lengthBA / ( lengthBA + lengthCA );
            vec3 D = B.xyz + ratio * CB.xyz;
            vec3 AD = D - A.xyz;
            vec3 bisect = normalize(AD);

            float theta = acos( dot(BA, CA) / (lengthBA * lengthCA) ) / 2.0;
            float AE = 1.0/(sin(theta)*aspect);
            newPos.z += AE/length(AD) * (D.z - A.z);
            newPos.x += bisect.x*AE;
            newPos.y += bisect.y*AE;

            gl_Position = newPos;
        }


This code gives us the results below.



Note, there are a few edge-cases having to do with almost backface-culled triangles being flipped by this process, and I began to address this in code, yet decided to simply avoid these cases for now. Perhaps I'll revisit it when I get this project done.