How can one estimate the difficulty change?
You cannot predict the change in difficulty because that is a function of a number of factors that are completely unknowable.

For example, the difficulty changes because a large number of mining pools suddenly appear and have very powerful ASICs and a lot of them. How would you ever be able to "predict" this happening?

Another example is the opposite case, that the technology has reached it's absolute maximum and there are no new chips yet invented that could give performance boost. Although new miners enter the market, this does not fundamentally change the rate at which the new blocks are discovered, because fundamentally the chips being used are no more powerful.

How on earth would you be able to determine that this has happened or is occurring?

At best any calculator would be guessing.