What is the importance of Modular arithmetic in cryptography? Why do we use modular arithmetic so often in Cryptography?
A few reasons:


As mentioned, modular arithmetic allows groups. See @mikeazo's answer.
Cryptography requires hard problems. Some problems become hard with modular arithmetic. For example, logarithms are easy to compute over all integers (and reals), but can become hard to compute when you introduce a modular reduction. Similarly with finding roots.
Cryptography is implemented digitally. It is nice if values can't be of arbitrary size. If you work with modular arithmetic, you have guarantees about the largest value you will see and can allocate the correct amount of space to hold values.