Why does the time complexity of RSA encryption and decryption depend on the size of the modulus?
The number of CPU cycles required to perform each multiplication increases with the size of the modulus $N$, since the operands can be as large as $N$, and also multiplication is followed by reduction modulo $N$.

Depending on what exactly you are analysing, you may sometimes consider $N$ to be constant (e.g., if you are comparing two exponentiation algorithms) and discard it when doing asymptotic complexity analysis, but if $N$ can vary, then certainly a larger $N$ means your computation will take more CPU cycles.