Curve rotation in GLSL fragment shader
You are doing completely different things in Matlab than in your Fragment Shaders.

In Matlab you are generating bunch of points that can be drawn anywhere on the screen and then you are transforming them. This would be loose equivalent of your Matlab code in Fragment Shader:

#define ARRAY_SIZE 256

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    st = st * 2.0 - 1.0;
    st.x *= u_resolution.x / u_resolution.y;

    //the angle to rotate:
    float ang = u_time * 0.5;

    //Transform coordinates:
    float c = cos(ang); 
    float s = sin(ang);    
    mat2 trans = mat2(c, s, -s, c);

    // our domain is from -1 to 1 so total length is 2.0
    float xStep = 2.0 / float(ARRAY_SIZE);

    // How far we are from current pixel location to the curve
    float minDist = 1e3;

    // initialize points
    for (int i = 0; i &lt; ARRAY_SIZE; ++i) {
        float x = -1.0 + float(i) * xStep;

        // initialize points
        vec2 cp = vec2(x, 0.08 * sin(x * 40.0));

        // transform
        cp = trans * cp;
        minDist = min(minDist, distance(st, cp));
    }

    // Add some contrast:
    float d = pow(minDist, 0.1);

    vec3 col = vec3(fract(d));
    gl_FragColor = vec4(col, 1.0);
}


(Here is the live version: http://thebookofshaders.com/edit.php?log=180516153813 ). 


For drawing I'm using distance to points because of simplicity, but better way would be to use distance to line segments that the points should be connected by.

In the other hands, in your fragment shaders you are first calculating something relative to a fixed pixel location (gl_FragCoord.xy), then transforming it in a way that it is not valid from the pixel's point of view any more, and then you are hoping that you can calculate proper distance to it.
Unfortunately this won't work.



(Example of fixed location of a Fragment Shader invocation)

But what you can do is first rotate the domain (and not result of your calculations), and then perform all calculations in it:

// Plot a line on Y using a value between 0.0-1.0
float plot(vec2 st, float pct){
  return  smoothstep( pct-0.02, pct, st.y) -
          smoothstep( pct, pct+0.02, st.y);
}

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    st = st * 2.0 - 1.0;
    st.x *= u_resolution.x / u_resolution.y;

    //the angle to rotate:
    float ang = u_time * 0.5;

    //Transform coordinates:
    float c = cos(ang); 
    float s = sin(ang);    
    mat2 trans = mat2(c, s, -s, c);

#if 1
   // You need to transform domain first:
   st *= trans;    

   // then perform the calculations:
   vec2 cp = vec2(st.x, 0.08 * sin(st.x * 40.0));
#else    
    // This won't work (your old way):
    vec2 cp = vec2(st.x, 0.08 * sin(st.x * 40.0));
    cp = trans * cp; 
#endif

    vec3 col = vec3(fract(distance(st, cp)));
    col.r += plot(st, cp.y);

    gl_FragColor = vec4(col, 1.0);
}


(Live version: http://thebookofshaders.com/edit.php?log=180516154712 )