Reign of Terror on Through the Ages
On page 12 of the rule book, it states the following


  When a card says you must "decrease"
  your population, you return 1 yellow
  token from your Unused Workers Pool.
  If you have no Unused Workers, you
  must remove (and return to your
  Yellow Bank) a Worker from one of your cards. Do not forget to adjust
  your indicators.


Therefore, you must lose a worker from your Pool (or one of your cards if no workers in your pool).