Does coffee in a cafe press keep brewing after being 'squeezed'?
The brewed coffee stays in contact with the ground, even though you pressed the sieve to the bottom. It will eventually release the unwanted flavors it contains, albeit slowly. The ground coffee should be in contact with the hot water for about 30 seconds.

See this answer for the caffeine content.