What prevents miners in a pool from retaining the reward?
The payout address for the miners mined blocks is the pool's moderator address. If the miners are working on blocks with a different payout address (ex. their own), the pool moderator can tell and act accordingly by either banning them or simply ignoring their work.