In the 1040s , the Zirids ( governors of North Africa under the Fatimids ) declared their independence from the Fatimids and their conversion to Sunni Islam , which led to the devastating BanÅ `` Hilal invasions . In the 1040s , the Berber Zirids ( governors of North Africa under the Fatimids ) declared their independence from the Fatimids and their recognition of the Sunni Abbasid caliphs of Baghdad , which led the Fatimids to launch devastating BanÅ `` Hilal invasions .