Why does the time complexity of RSA encryption and decryption depend on the size of the modulus? I would think that the time complexity would depend on the exponent, not the number of bits in the modulus. For example, if the exponent were 5, using the "naive" method, we would perform 5 multiplications. Squaring and multiplying would reduce the number of multiplications dependent on the exponent as well. However, I don't see where the number of bits in the modulus comes in to play.
The number of CPU cycles required to perform each multiplication increases with the size of the modulus $N$, since the operands can be as large as $N$, and also multiplication is followed by reduction modulo $N$.

Depending on what exactly you are analysing, you may sometimes consider $N$ to be constant (e.g., if you are comparing two exponentiation algorithms) and discard it when doing asymptotic complexity analysis, but if $N$ can vary, then certainly a larger $N$ means your computation will take more CPU cycles.