Why rectangular waveguides are more popular in RF/Microwave compare to circular waveguides?
This is because of mode propagation.  the highest possible bandwidth allowing only a single mode to propagate with circular waveguides is only 1.3601:1.

Rectangular waveguides have a much larger bandwidth over which only a single mode can propagate.

I suggest you try Wikipedia: https://en.wikipedia.org/wiki/Waveguide_(electromagnetism)