Why is the difficulty adjustment restricted to a factor of 4? What does this protect against? And why 4 in particular?

I know an adjustment of this magnitude is not likely to take place at this stage, but I'm wondering why this was implemented in the first place.
You need restrict how much difficulty can change. Imagine if someone managed to forge the timestamps on the block that they were creating, and make it so that it appeared that it only took one second to generate 2016 blocks. Should difficulty then go up by 1209600? Clearly, that would be ridiculous.

On the other hand, you need to make the adjustment flexible enough to adapt to unforeseen advances. 4 is an arbitrary value picked as a compromise between these two requirements.