Derivation of this mho circle equation?
I'm looking in the old version (yellow cover) and it defines VLN as line-to-line in the text underneath the equation.  One of the later revisions injected a bunch of errors (can't remember which one as i stick with old yellow).  You have found one such error and you are correct.  Equation 6.3 is only correct when you recognize VLN is line-line voltage.  Sloppy work in this edition.
Just use equation 6.5 to plot the circle.  ZR is the reach at maximum torque angle (φr) and φx is the angle at which you are finding a point on the circle (ZX).
If i recall correctly, you can find out if your version of the text is the garbage version by going to the list of device numbers.  Many of them were wrong (i think including device 21 which should be distance relay).