However, the resolution is non-binding, with Nigeria and Senegal opposing a withdrawal.
South Africa and Burundi have already decided to withdraw, accusing the ICC of undermining their sovereignty and unfairly targeting Africans.
The ICC denies the allegation, insisting it is pursuing justice for victims of war crimes in Africa.
The AU took the decision on Tuesday following a divisive debate at its annual heads of state of summit in Addis Ababa.
Part of the resolution also said the AU would hold talks with the UN Security Council to push for the ICC to be reformed.
After being discussed in several previous summits, this was a huge announcement showing how frustrated the AU was with the international court. But the debate itself showed how divisive the whole issue is.
The resolution isn't as strong as many who are opposed to the court would have liked. It only calls on countries to consider how to implement the decision but does not bind them to it. It's a victory for human rights activists who insist the court still has a very important role to play in the continent where many countries have weak judicial systems.
The resolution also calls for African countries to continue pushing for reforms of the court - another clear indication that ditching the court en masse isn't such a popular decision. The likes of South Africa and Kenya, which have pushed for withdrawing, will be disappointed that the discussions about completely severing ties with the ICC will have to wait another six months for the next summit.
Sudan's President Omar al-Bashir, who is wanted by the court on charges of genocide in Darfur, was at the summit.
In 2015, a South African court criticised President Jacob Zuma's government for failing to arrest Mr Bashir when he attended an AU meeting in the main city, Johannesburg.
The government later announced that it was withdrawing from the ICC because it did not want to execute arrest warrants which would lead to "regime change".
A total of 34 African states are signatories to the Rome Statute, which set up the ICC.
The ICC and global justice:
What does the ICC do?
The African Union has called for the mass withdrawal of member states from the International Criminal Court (ICC).