Curve rotation in GLSL fragment shader I'm trying to figure out the correct math to rotate and translate a curve displayed in a fragment shader.

What I try to accomplish is to define a curve, for example a sine curve, in a local coordinate system, rotate it then translate it. Something like this:



That was made in MATLAB with the following code:

dens = 1080;
x = linspace(-1.0, 1.0, dens);        
y = 0.1*sin(25.0*x);

imax = 25;
for i = 1:imax    

    %transformation matrix:
    ang = (i/imax)*0.5*3.14;
    c = cos(ang); s = sin(ang);          
    T = [c,-s;s,c];

    %translation:
    P = [0.5;0.5];

    %transformed coordinates:
    xt = T(1,:)*[x;y] + P(1);
    yt = T(2,:)*[x;y] + P(2);

    plot(xt,yt);
    xlim([0 1.0]); ylim([0 1.0]); drawnow;
end


For the GLSL test I'm using the Book of Shaders Editor with the following code (can also be seen interactively here): 

#ifdef GL_FRAGMENT_PRECISION_HIGH
precision highp float;
#else
precision mediump float;
#endif

uniform float u_time;
uniform vec2 u_resolution;

// Plot a line on Y using a value between 0.0-1.0
float plot(vec2 st, float pct){
  return  smoothstep( pct-0.02, pct, st.y) -
          smoothstep( pct, pct+0.02, st.y);
}

float plotTransformed(vec2 st, float pct, vec2 transl, float ang){

    float c = cos(ang); float s = sin(ang);    
    mat2 trans = mat2(c,-s,s,c);    
    st = trans * st;

    st -= transl;

    return  smoothstep( pct-0.02, pct, st.y) -
          smoothstep( pct, pct+0.02, st.y);
}

void main(void) {
    bool use_plot_function = true;

    float mx =  max(u_resolution.x, u_resolution.y);
    vec2 uv = gl_FragCoord.xy /mx;
    vec3 color = vec3(0.4,0.4,0.4);

    //some screen position:
    vec2 p = vec2(0.5, 0.5);

    //the curve:
    vec2 cp = vec2(
        uv.x,
        0.08*sin(uv.x*40.0)
    );

    //the angle to rotate:
    float ang = -0.4 * 3.14 * sin(u_time);

    //Transform coordinates:
    float c = cos(ang); float s = sin(ang);    
    mat2 trans = mat2(c,-s,s,c);    
    vec2 cp_t = trans * cp;    
    cp_t +=p;



    if(use_plot_function){
        //Attempt 1: plot unrotated original curve translated upwards: 
        float curve1 = plot(uv, cp.y + p.y);
        color.g *= curve1;    

        //Attemp 2: plot the transformed curve using plotTransformed, rotates first, then translates:
        float curve2 = plotTransformed(uv, cp.y, p, ang);
        color.r *= curve2;

        //Attempt 3: curve is transformed first then ploted:
        float curve3 = plot(uv, cp_t.y);
        color.b *= curve3;
    }            
    else{
        float plotThk = 0.02;

         //Attempt 1: change color based on distance from unrotated original curve: 
        float dist = distance(uv, cp + vec2(0.0, p.y));
        if(dist &lt; plotThk)
            color.g *= (1.0 -dist)/plotThk;   

        //Attempt 2: change color based on distance from transformed coordinates:
        dist = distance(uv, cp_t);
        if(dist &lt; plotThk)
            color.r *= (1.0 -dist)/plotThk;   

    }

    gl_FragColor = vec4(color,1.0);
}


In the code above, there are two modes which can be toggled with use_plot_function set to false or true.

First mode attempts to plot using the functions plot() &amp; plotTransformed().
Second mode sets a color to a fragment based on the distance from the calculated curve coordinates.

Result of first mode with use_plot_function set to true:



Result of second mode with use_plot_function set to false:



Obviously I'm misunderstanding how this should be done in a fragment shader. 

How should I correctly define a transformed curve in GLSL fragment shader?
You are doing completely different things in Matlab than in your Fragment Shaders.

In Matlab you are generating bunch of points that can be drawn anywhere on the screen and then you are transforming them. This would be loose equivalent of your Matlab code in Fragment Shader:

#define ARRAY_SIZE 256

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    st = st * 2.0 - 1.0;
    st.x *= u_resolution.x / u_resolution.y;

    //the angle to rotate:
    float ang = u_time * 0.5;

    //Transform coordinates:
    float c = cos(ang); 
    float s = sin(ang);    
    mat2 trans = mat2(c, s, -s, c);

    // our domain is from -1 to 1 so total length is 2.0
    float xStep = 2.0 / float(ARRAY_SIZE);

    // How far we are from current pixel location to the curve
    float minDist = 1e3;

    // initialize points
    for (int i = 0; i &lt; ARRAY_SIZE; ++i) {
        float x = -1.0 + float(i) * xStep;

        // initialize points
        vec2 cp = vec2(x, 0.08 * sin(x * 40.0));

        // transform
        cp = trans * cp;
        minDist = min(minDist, distance(st, cp));
    }

    // Add some contrast:
    float d = pow(minDist, 0.1);

    vec3 col = vec3(fract(d));
    gl_FragColor = vec4(col, 1.0);
}


(Here is the live version: http://thebookofshaders.com/edit.php?log=180516153813 ). 


For drawing I'm using distance to points because of simplicity, but better way would be to use distance to line segments that the points should be connected by.

In the other hands, in your fragment shaders you are first calculating something relative to a fixed pixel location (gl_FragCoord.xy), then transforming it in a way that it is not valid from the pixel's point of view any more, and then you are hoping that you can calculate proper distance to it.
Unfortunately this won't work.



(Example of fixed location of a Fragment Shader invocation)

But what you can do is first rotate the domain (and not result of your calculations), and then perform all calculations in it:

// Plot a line on Y using a value between 0.0-1.0
float plot(vec2 st, float pct){
  return  smoothstep( pct-0.02, pct, st.y) -
          smoothstep( pct, pct+0.02, st.y);
}

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    st = st * 2.0 - 1.0;
    st.x *= u_resolution.x / u_resolution.y;

    //the angle to rotate:
    float ang = u_time * 0.5;

    //Transform coordinates:
    float c = cos(ang); 
    float s = sin(ang);    
    mat2 trans = mat2(c, s, -s, c);

#if 1
   // You need to transform domain first:
   st *= trans;    

   // then perform the calculations:
   vec2 cp = vec2(st.x, 0.08 * sin(st.x * 40.0));
#else    
    // This won't work (your old way):
    vec2 cp = vec2(st.x, 0.08 * sin(st.x * 40.0));
    cp = trans * cp; 
#endif

    vec3 col = vec3(fract(distance(st, cp)));
    col.r += plot(st, cp.y);

    gl_FragColor = vec4(col, 1.0);
}


(Live version: http://thebookofshaders.com/edit.php?log=180516154712 )