Most historical Muslim rulers simply titled themselves sultans or amirs , and the caliph himself often had very little real authority . Most historical Muslim governors were called sultans or emirs , and gave allegiance to a caliph , but this caliph at times had very little real authority .