Derby is one of only four British Prime Ministers to have been in office three or more times . His record was unusual , since he is one of only four British Prime Ministers to have three or more separate periods in office .