How was Madara so sure that he would be able to get his Rinnegan back?
As Madara himself said his plan was to be resurrected through the use of Rinne Rebirth by Nagato. He was surprised and disappointed that his return didn't go as planned. He didn't want the Edo Tensei, as only a living person can become the Ten-Tails' jinchūriki.

Considering that the usage of the Rinne Rebirth always comes at the cost of the user's life it wouldn't be a problem for Madara to then take the Rinnegan from Nagato's corpse.

How exactly they wanted to force Nagato to resurrect Madara is unknown. It could be simply Obito convincing him that it would be the necessary next step in their plan towards world peace or by forcing him to use it via Genjutsu or a Zetsu takeover. 


  But if Black Zetsu can overpower a Ninja as powerful as Nagato could he not have simply overpowered Hagoromo at the very beginning and be done with it?


Nagato was far less powerful than Hagoromo. It pushed Nagato to the brink to trap Naruto inside his Chibaku Tensei, which was - as he told Konan - nothing compared to the creation of the moon by Hagoromo.