"In full co-ordination with the government of Iraq, we're deploying a specialised expeditionary targeting force to assist Iraqi and Kurdish Peshmerga forces," Mr Carter said.
The operators would carry out raids, free hostages and capture IS leaders.
They would also conduct "unilateral operations" in Syria, he said.
Islamic State conquered large swathes of Iraq and Syria in the summer of 2014.
A US-led coalition of 65 countries is conducting a campaign against the militants in both countries.
Mr Carter made the announcement in remarks to the House Armed Services Committee in Washington on Tuesday.
He did not offer specifics on troop numbers, but said they would also be focused on securing Iraq's borders and building the capacity of the Iraqi government forces.
US special forces have already conducted operations in both Syria and Iraq - notably the killing of the IS oil chief Abu Sayaff in eastern Syria and the freeing of prisoners from the IS-held town of Hawija near Kirkuk in Iraq.
But this is a clear indication from the Pentagon that their role will become more significant.
The "specialised expeditionary targeting force", as the Iraq group is being called, will not only assist local forces but will also conduct further "unilateral" raids into Syria.
While this does not amount to significant numbers of boots on the ground, it is an escalation of America's role in direct combat.
Mr Carter said there was now an important opportunity to divide IS militants in Iraq from those in Syria.
In October, the US announced it would send up to 50 special forces soldiers to northern Syria as military advisers to Kurdish and rebel forces.
The US is to deploy a specialised force to Iraq to build pressure on Islamic State militants, US Defence Secretary Ashton Carter has said.