A statement said the drive was aimed at "asserting the nation's territorial integrity" and "enhancing security".
On Tuesday President Goodluck Jonathan declared a state of emergency in Yobe, Borno and Adamawa states after a series of deadly attacks by militants.
Fighters from the Boko Haram group are blamed for most of the violence.
The group, which has its roots in north-eastern Nigeria, is linked to bloodshed in which some 2,000 people have died since 2010.
Boko Haram has seized control of parts of the north-east over the past three years, and much of the violence has been confined to that region.
By Will RossBBC Nigeria correspondent
There is no doubt that the situation in northern Nigeria is deteriorating fast and now President Goodluck Jonathan has admitted as much. No-one else in the government had dared say a word.
The frequency and scale of the attacks carried out by Boko Haram have forced the president to take action and to silence some of his critics who accuse him of being out of touch and slow to react.
Some analysts have suggested that if left to fester there is little to stop Boko Haram from setting up its own Islamic institutions in the extreme north-east.
Will more boots on the ground improve the situation? Much will depend on how the troops behave. Analysts argue that the military has already lost the battle for hearts and minds and residents feel trapped between the two sides of this murky conflict.
Can Nigeria quell Boko Haram?
However, the group did bomb Nigeria's capital Abuja in 2011 and 2012.
The militants have forged closer links with al-Qaeda-linked militants in the Sahara region and have access to increasingly sophisticated weaponry. They are believed to move back and forth across frontiers, particularly the Cameroon border, say analysts.
Nigeria, a multi-ethnic nation of more than 160 million people, is also affected by conflicts over land, religion and oil.
Wednesday's Ministry of Defence statement said the army, police and other security agencies had begun operations to "rid the nation's border territories of terrorist bases and activities".
The declared aim that it will assert the nation's territorial integrity comes a day after the president said the state no longer controlled the entire territory of Nigeria, a somewhat embarrassing admission, says the BBC's Will Ross in Lagos.
The operation will focus on enhancing security of governmental structures after the president said attacks on government buildings and killings of officials and other civilians amounted to a declaration of war.
Borno, Adamawa, and Yobe are three of the poorest states, far removed from the southern oilfields that feed government coffers.
"We've had a lot of problems [of] border crime, and criss-crossing of the border by the insurgents, and there's also evidence that some of the insurgents really are non-Nigerians," presidential spokesman Doyin Okupe told the BBC.
From preachers to slave raiders
"As long as the terrorists can go in and out unchallenged, then we're in big trouble."
Residents and reporters in the north-east said there had already been an influx of extra troops and military hardware.
"I have never seen soldiers on the move quite like this before," said Ahmed Mari, who lives in Maiduguri, the capital of Borno state.
"A lot of fighting equipment has been deployed to Maiduguri," Hajja Maimuna, whose house is close to Maiduguri's Air Force base, told Nigeria's Daily Trust website.
"We are really terrified... Even though the state governor has not been removed, which is good, the truth is that Maiduguri is now a garrison town, full of angry soldiers who are ready to kill at the slightest provocation."
Boko Haram, whose name means "Western education is forbidden" in the local Hausa language, is fighting to overthrow the government and create an Islamic state in the north.
Although they often attack Christians and government targets, they have also killed many Muslim civilians.
Nigeria has sent a "massive deployment of men and resources" to combat Islamist militants in three north-eastern states.