The All Progressives Congress (APC) won 19 of the 28 governor posts in results declared from Saturday's elections.
It is the biggest defeat for the Peoples Democratic Party (PDP) since military ruled ended in 1999.
It lost the presidential poll last month for the first time.
APC leader Muhammadu Buhari will be inaugurated on 29 May, after he defeated President Goodluck Jonathan.
The PDP's decline has been reinforced by results from Saturday's elections for governors and state assemblies, reports the BBC's Bashir Saad Abdullahi from the capital, Abuja.
The party lost for the first time in key northern states like Katsina and Kaduna, the home of outgoing Vice-President Namadi Sambo.
It also lost control of Adamawa, one of three states badly hit by militant Islamist group Boko Haram's insurgency.
However, the party managed to hold on to Rivers State, Nigeria's oil hub.
The PDP won eight governorships, mostly in the south-east.
Taraba is the only result still outstanding, while the election in Imo state was declared inconclusive, because the number of spoilt ballot papers was larger than the margin of victory.
Elections were not held in seven states.
Analysis: Chris Ewokor, BBC Africa, Abuja
The APC has made historic gains, relegating the once-powerful PDP to a regional party. The PDP failed to get a national spread of votes, doing well only in the south-east and the oil-rich Niger Delta. Most states in the north went to the APC, along with Nigeria's commercial hub, Lagos, in the south-west.
However, the APC failed to win in any of Nigeria's five oil-producing states. This will be of huge concern to President-elect Muhammadu Buhari, as militant groups could disrupt oil production and starve the federal government of much-needed oil revenue.  It is unlikely that the PDP governors in the five states would help Gen Buhari rein in the militants.
In parts of the country, the elections were marred by violence due to stiff rivalry, and the battle for power and money. Ethnic differences also fuelled the conflict in some areas, but overall the elections have been more peaceful than in previous years, raising hopes that Nigeria's democracy is maturing.
Why Jonathan lost?
Buhari's historic victory
Nigeria's 36 governors enjoy wide powers and are extremely influential. Some, especially in oil-producing areas, control bigger budgets than those of national governments in some neighbouring West African countries.
The Independent National Electoral Commission (Inec) reported 66 violent incidents at polling stations - higher than in the presidential poll.
Most of the violence took place in Rivers and other southern states, where voting at some polling stations had to be extended.
On Sunday, Inec also announced that the APC had retained power in Lagos state.
It means that when Gen Buhari takes over the presidency it will be the first time that the party of the president has controlled Lagos - the commercial capital - since the advent of multi-party democracy in 1999.
Our correspondent says the PDP will have to go through a period of introspection, and then rebuild itself under a new leader.
The party was in power for 16 years, and voters felt that it was time for change, he says.
It had been hit by the defection of governors and MPs to the APC ahead of the elections.
The PDP also faced widespread allegations of corruption during its rule, and was accused of failing to do enough to end an Islamist-led insurgency in the north-east where it suffered huge defeats.
The party of Nigeria's incoming president has won a landslide in elections for powerful state governors, ending the former ruling party's dominance.