Calculating angle in isometric view
@neo pointed out that dx can be calculated as (1 - r) * Math.cos(a - b).  Similarly, dy = (1 - r) * Math.sin(a).  However, when you take dy / dx, which is what Math.atan2(dy, dx) implicitly does with its arguments, the (1 - r) factor cancels out.  You can also see, using a geometric argument, that (x2, y2) is pointless (pardon the pun).



Therefore, Math.atan2(y1, x1) would work just the same as Math.atan2(dy, dx).



So far, your function can be simplified to the following.  Since you repurposed angle twice, I need to disambiguate them as alpha, beta, and theta for this discussion.

public static inline var ISO:Float = 0.45378560551; // (116-90) / 180 * PI=;

function convert(alpha:Float):Float
{ 
  var beta = alpha - Math.PI; // corrected angle 

  // calculate new line, using isometic angle.
  var x1 = Math.cos(beta - ISO);
  var y1 = Math.sin(beta);

  var theta = Math.atan2(y1, x1);
  return theta;
}




But wait, there's more!  There's a mysterious correction from alpha to beta, and the cosine expression is complicated.

Let's start with Math.sin(beta).  That's Math.sin(-alpha), or -Math.sin(alpha).

It would be nice to say Math.atan2(Math.sin(alpha), something) instead of Math.atan2(-Math.sin(alpha), something).  Let's move the negation into the denominator then, for var theta = Math.atan2(Math.sin(alpha), -x1).

Can we simplify -x1?

-x1 = -Math.cos(beta - ISO)
    = Math.cos(beta - ISO + 180°)
    = Math.cos(alpha - 180° - ISO + 180°)
    = Math.cos(alpha - ISO)


So, your function simplifies to:

public static inline var ISO:Float = 0.45378560551; // = (116-90) / 180 * PI

function convert(angle:Float):Float
{ 
    return Math.atan2(Math.sin(angle), Math.cos(angle - ISO));
}


Not only is the code more efficient, it's also less mysterious: the transformation is taking the x-coordinate of each point as if it were rotated 26° clockwise!



I have to question the motivation behind this function, though.  This isn't an angle-preserving transformation, so why are you operating on an angle?  Normally, you transform points' coordinates using matrix multiplication.