If we could actually compute the key this way, well then the protocol would be completely unsecure because it turns out that modular exponentiation is a function that we can compute efficiently, and it's necessary that we can compute modular exponentiation efficiently because otherwise, the legitimate participates in the protocol wouldn't be able to determine their keys. 