In the shor's quantum circuit, what is the transformation of the modular operating gates?
It's a modular multiplication gate. It permutes the states such that, for all $k$ below $N$, state $|k\rangle$ becomes state $|k \cdot c \pmod{N}\rangle$ where $N$ is the number being factored and $c = b^{2^{2n-1}} \pmod{N}$ is a classically computed constant and $b$ is the base value chosen randomly at the start of the algorithm.