The episode "It's Your Funeral" suggests that most of the Number Twos encountered in the series are in fact only temporary appointments, standing in for an older Number Two who has been away from the Village for reasons not explained.
Exactly who operates the Village is deliberately obscured.