How is difficulty calculated? Can anyone explain me in the plain English how difficulty is calculated. I have a very approximate understanding that it is calculated based on the amount of hash power in all the bitcoin community over a specific period of time. But this is very vague.

Also I understand it can change very rapidly. Can it only increase? Is there any formula how to calculate it or predict it?

Thanks for a detailed answer, Meni Rosenfeld. Just to make sure I got everything right. I am summing up all the time, it took to generate the last 2016 blocks. And then apply the formula.
The Bitcoin difficulty started at 1 (and can never go below that). Then for every 2016 blocks that are found, the timestamps of the blocks are compared to find out how much time it took to find 2016 blocks, call it T. We want 2016 blocks to take 2 weeks, so if T is different, we multiply the difficulty by (2 weeks / T) - this way, if the hashrate continues the way it was, it will now take 2 weeks to find 2016 blocks.

For example, if it took only 10 days it means difficulty is too low and thus will be increased by 40%.

The difficulty can increase or decrease depending on whether it took less or more than 2 weeks to find 2016 blocks. Generally, the difficulty will decrease after the network hashrate drops.

If the correction factor is greater than 4 (or less than 1/4), then 4 or 1/4 are used instead, to prevent the change to be too abrupt.

There is a bug in the implementation, due to which the calculation is based on the time to find the last 2015 blocks rather than 2016. Fixing it would require a hard fork and is thus deferred for now.

It is possible to give a rough estimate for the next difficulty change, based on the time to find the recent blocks. Nobody can make longer-term predictions for the future difficulty reliably, but anyone is free to speculate based on exchange rate trends, Moore's law and other hardware advances.