What does this symbol mean? And I have some basic problems which the book didn't mention
The symbol is of a current transformer, or more correctly, 2 wired in anti-parallel, 
As they are wired in opposite directions, if they had the same level of current they would cancel each other out, as such you end up with the difference in current between the 2 coils, IR - IT, There is no transformer measuring IS, so its not part of the math
Without knowing the context, I would say your safe to substitute with I1,I2,I3, It appears to just be transformer phases
Best refer you to here, the square root of 3 is a quirk of 3 phase systems when measuring between 2 phases, Square Root of 3