Why are these techniques not feasible to crack RSA? For $d = e^{-1} \mod \phi(n)$

$$d\cdot e \operatorname{mod} \phi(n)=1$$
$$d = (e^{-1} \operatorname{mod} \phi(n))$$

With having $e$ and $n$, we can calculate the value of $d$ , the private key for cracking.
Why do people worry about factoring $d$ when this technique is available?
I would like to know some real insight into this.
You are correct in that knowing $\phi(n)$ it is trivial to get the private key back with a simple modular inversion. 

However, we are only given $e$ and $n$, and it turns out that computing $\phi(n)$ from $n$ alone is computationally equivalent to finding the factors of $n$. Namely, if you know $\phi(n) = (p-1)(q-1) = (p-1)(n/p - 1)$, you can recover $p$ by solving the quadratic equation $p^2 + (\phi(n) - n - 1)p + n = 0$ for $p$.