The watchdog's annual report on education standards says about a third of secondary schools in the North and Midlands are not good enough.
Ofsted chief Sir Michael Wilshaw says more attention must be focused on regions where too many schools are "languishing in mediocrity".
Education Secretary Nicky Morgan said more needed to be done to "tackle pockets of underperformance".
The chief inspector of schools warns of schools in some cities in the North and Midlands that have "failed miserably year after year after year".
Ofsted annual report 2015
Sir Michael named 16 weak local authorities - where less than 60% of children attend good or outstanding schools and have below average "attainment and progress" at GCSE - with 13 in the North and Midlands.
They are Barnsley, Blackpool, Bradford, Derbyshire, Doncaster, Hartlepool, Knowsley, Liverpool, Middlesbrough, Oldham, Salford, St Helens and Stoke on Trent.
The other three are the Isle of Wight, Swindon and South Gloucestershire.
In Bradford, Sir Michael said the problems were so serious that he wanted a commission of inquiry and a separate commissioner for education.
And he warned that poor education meant that young people could be alienated and the "dangers of being alienated from British society are very great indeed".
Increasing attention is being paid to the geography of school underachievement - such as in coastal areas and regional differences.
And the report highlights concerns about a north-south gap - with stronger GCSE results in London and the South and weaker results in parts of the North and Midlands.
"This gap is a worrying one. We don't want to see a divided country after the age of 11," Sir Michael said.
More than 400,000 pupils in the North and Midlands go to a school that is "less than good", and the Ofsted chief said this could not be "explained away" by higher levels of deprivation.
Primary schools were much more successful than secondary, Sir Michael said, despite facing similar problems of social disadvantage.
He also used the example of London to show how deprivation need not be inevitably linked to lower results.
Sir Michael called for greater "political will" and more attention to improve the quality of teaching.
"We've seen a significant difference in the quality of teaching between the South and the Midlands and the North, a significant difference in terms of the quality of leadership... and we need to worry about this as a nation," he said.
"The mediocrity in secondary performance should be a national concern - and the mediocrity is residing mainly, but not exclusively, in the Midlands and north of England.
"If the 'northern powerhouse' is going to mean something, then we've got to improve educational performance in our secondary schools north of the Wash."
Sir Michael also warned that problems were being exacerbated by difficulties in recruiting teachers - and that schools in challenging areas could have the toughest time finding enough teachers in some subjects.
Recruitment problems were having a "significant impact", said the Ofsted chief.
Watchsted, a website that gathers inspections data, ranks local authorities in terms of the proportion of schools rated as "outstanding" or "good".
The bottom 10 at secondary school level includes northern authorities such as Knowsley, Bradford, Oldham, Blackpool and Doncaster - but also the Isle of Wight and Cambridgeshire.
Eight out of the top 10 authorities at secondary level are in London or the South East.
"More and more we are seeing that schools facing the biggest challenges are now in coastal and market towns, rather than large cities. And children from low income families have the greatest barriers," said James Westhead, executive director of Teach First.
Most secondary schools in England now have academy status - but the annual report says that in terms of raising standards "structural reform can only do so much".
And it highlights that being an academy does not "insulate" schools from decline, with 99 academies that had been good or outstanding being regraded as less than good.
Roy Perry, chairman of the children and young people board at the Local Government Association, said ranking councils by school results "disguises the fact that over 60% of secondary schools are now academies, leaving councils powerless to intervene early and build an improvement programme".
"It is extremely worrying that over the last three years only 37% of secondary schools have actually improved their Ofsted rating after becoming academies."
The Ofsted report also highlights that pupils' progress up to the end of primary is too often not sustained when they move to secondary.
This was revealed last month in an analysis of this summer's GCSE results, by education data company, SchoolDash.
This mapped the growing north-south divide at GCSE level, but also showed there was much less of a gap in primary school.
Head teachers' leader Brian Lightman linked the north-south gap to the "very serious difficulties" that schools faced in recruiting teachers.
Mr Lightman, head of the ASCL head teachers' union, said schools in challenging areas were finding it "incredibly hard" to find teachers in some shortage subjects, particularly affecting schools in some parts of the North.
Russell Hobby, leader of the National Association of Head Teachers, said the gap between primary and secondary performance had been "driven by the turmoil at secondary compared to primary", such as the upheavals in exams and curriculum.
Education Secretary Nicky Morgan said: "More needs to be done to deliver educational excellence everywhere."
"That's why we are introducing new measures to transform failing and coasting schools, funding the best academy chains to share excellence in struggling regions in the North and creating a National Teaching Service - sending some of our best teachers to the areas that need them most."
Ofsted is warning of a north-south divide in England's secondary schools.