Why is RSA encryption key based on modulo $\varphi(n)$ rather than modulo $n$? While calculating RSA encryption key we take modulo $\varphi(n)$ rather than modulo $n$. I can’t understand why it’s done this way.
Firstly, a number theory textbook might well help here. I own An Introduction to the theory of numbers which will give you a grounding in this and many other number theory topics which crop up in crypto. It's not my favourite book notation-wise, but in terms of topics covered it is very thorough.

Now, on to what you need help with. There exists a theorem called the division theorem that says any number $n = pq + r$, with the constraints that $n &gt; q &gt; r \geq 0$. In plain words, any number $n$ may be represented by a quotient $q$ (possibly multiplied by a number $p$) and a remainder $r$. An example would be to say that $13 = 7 + 6$ - you recognise this in that $13/7$ could be written as "$1$ remainder $6$".

Modular arithmetic is related - let's start off with $13 \equiv 6 \mod 7$. Intuitively, modular arithmetic is often explained as "you take the remainder and that is the result" which isn't far from the truth - what we actually say is that $7$ and $13$ are part of the same congruence class - that is, $q$ and $r$ ($7$ and $6$) remain the same and the congruence class is made up of all the $n$ for all the $p$ possible. So for $p=2$, $n = 2*7 + 5 = 19$. So $19\equiv 5 \mod 7$ (feel free to check it!).

So, you're concerned with the use of $\mod \phi(n)$ in RSA - well the expression usually given is $de = 1 \mod \phi(n)$. We can re-write that in division-theorem form as $de = k\phi(n) + 1$ - where $k$ is $p$ from the previous example, but renamed so as not to be confused with the prime $p$ that forms $n$.

Now observe what you may have learnt about exponents - that $x^{a+b} = x^ax^b$ and that $x^{ab} = (x^a)^b)$ If you're not convinced, pick numeric examples. Now, a message $m$ raised to the power $de$ is $m^{de}$ - but we have a different expression for $de$ already - that is - $m^{de} = m^{k\phi(n) + 1}$. This can then be written as $(m^{\phi(n)})^k \times m^1$. 

There is a very good reason for this form being desirable - namely the Euler-Fermat generalisation to Fermat's Little Theorem, which states that:

$$a^{\phi(n)} = 1 \mod n$$.

Hopefully you should see that our manipulating the algebra to get $m^{\phi(n)}$ in it is not an accident at all - since under $\mod n$, $(m^{\phi(n)})^k \times m^1$ becomes $1^k \times m^1 = m$. This is what makes RSA work the way it does.

I'd also like to address the point fgrieu has raised in his comment, since it is important. The requirement is for the private key is such that $de = 1 \mod \phi(n)$ - or $de = 1 + k\phi(n)$. Clearly, $k$ can be any number we like - any congruence class for which this relation holds will work. 

The lowest common multiple of two numbers is the smallest possible number such that both these numbers divide it - for example, the lowest common multiple of $4$ and $6$ is $12$. n this case we observe $4$ divides $12$, as does $6$. Now, the lowest common multiple of $p-1, q-1$ is a number such that both $p-1$ and $q-1$ divide it - well $\phi(n)$ is one such number, for starters! However, (thanks to Poncho), we can say that it is not the smallest possible number. To explain this, consider that $p$ is an odd number and that odd numbers can all be written of the form $2k+1$ for some $k$. If we subtract one, we are left with an even number which will have at least one factor of $2$. Therefore, since $p-1$ and $q-1$ are both even and both have a factor of two, their product ($\phi(pq)$) will have two factors of $2$ at the least, which is 2x what is necessary (at the minimum). More generally, the definition of a lowest common multiple uses the fundamental theorem of arithmetic to demonstrate that the lcm is in fact equal to the product of the maximum power each prime is raised to in each number's prime expansion - eurgh, yes? Well, let's look at that. I chose $4$ and $6$ before - now the prime expansions are $4 = 2^2$ and $6=2\times 3$. Taking the maximum exponentiation of each prime, $2^2 \times 3 = 12$. However, if we did $4\times6 = 2^2 \times 2 \times 3 = 2^3 \times 3$ we see we have gained an extra, unnecessary factor of $2$.

It turns out the Euler-Fermat generalisation can be generalised again - to the Carmichael theorem which states that:

$$a^{\lambda(n)} = 1 \mod n$$

Where $\lambda(n)$ is the smallest such integer where this expression is true. A slight detour - $\phi(pq)=\phi(p)\phi(q)$ and since $\phi(p) = p-1$ that is how we end up with our definition of $\phi(pq)=(p-1)(q-1)$. Now, in the general case, it turns out that $\lambda(pq) = lcm(\lambda(p), \lambda(q)) = lcm(p-1, q-1)$.  By the same reasoning as the case with $\phi(pq)$, $de = 1 \mod \lambda(pq)$ allows us to invert $m^e$ back to $m$ under $\mod n$.

fgrieu has used the term "Rings" - if you're interested in those, I highly recommend Rings, Fields and Groups which is possibly the best Maths textbook I have ever read. It's about (abstract) algebra and also covers some of the number theoretic points, but not as many as a pure number-theory textbook does.