The agreement appears to bring to an end a bitter dispute between the coalition and the Afghan government.
The troops and their Afghan counterparts have been accused of murder and intimidation in the area.
Special forces and local police will now begin a phased withdrawal.
Afghan security forces will then take over on a district-by-district basis.
American special forces and the Afghan local police they have trained in Wardak have been accused of committing atrocities in the area, including the murder of nine men.
Isaf has denied the allegations but President Hamid Karzai last month said they should withdraw within a fortnight.
Relations between the president and Nato have been tense in recent weeks.
On Tuesday the presidential spokesman described the Nato-led military operation in Afghanistan as "aimless and unwise".
"We have come to agreement on a plan for Wardak that continues the transition of this critical province and meets the security needs of the people and the requirements of our mission," Isaf Commander Gen Joseph Dunford said.
"Under the agreement, beginning with Nerkh District, which is currently secured by Afghan Local Police (ALP) aided by coalition forces, the Afghan government will soon move Afghan National Security Forces (ANSF) into this area to provide security.
"The arrival of the ANSF will preclude the need for ALP and coalition forces in this area. The remainder of the province will transition over time."
The general said that the agreement reflected the "growing capacity and capability of the Afghan security forces to meet the security needs" of Afghanistan.
The BBC's Quentin Sommerville in Kabul says that the agreement is short on specifics - no timetable has been reached on when the special forces will leave. But it does allow Mr Karzai to save face.
Afghan defence ministry spokesman Gen Zahir Azimi said that the Afghan army "will take the lead" after international special forces withdraw from Nerkh district, "so this problem will be solved within a few days".
However our correspondent says that it is far from certain that the already stretched Afghan special forces will be able to establish security in Wardak once Nato has left.
Wardak is a critical province. Located near Kabul, it has been used as staging post for a number of large-scale attacks in the city and has been the recent focus of counter-insurgency operations.
The accountability of US forces and local militia working with them has been a growing source of friction in Afghan-US relations.
Wednesday's agreement comes against a backdrop of long-term negotiations over which foreign forces will remain in Afghanistan after Nato's exit in 2014.
The bulk of Nato's 100,000 troops are due to leave by the end of that year.
An agreement has been reached between Nato and the Afghan government on the withdrawal of US special forces from Wardak province, alliance officials have announced.