He told MPs the UK would never be a world-beating economy unless young people were better educated.
And he called for "much bigger devolution to the people who know where the [failing] schools are".
On improving literacy and numeracy, he added: "We need it across the country and we need it yesterday."
The Conservative grandee, who was deputy prime minister in the 1990s and subsequently advised David Cameron on regional strategy, made the comments as he gave evidence to the Commons Business Innovation and Skills Committee alongside former chancellor George Osborne.
Claiming that the UK is "29th as a country in the world league of education" - it was 23rd in reading in 2015, according to the OECD's Pisa rankings and 26th in maths - he stressed: "If we accept that - we'll never be a world beating economy in the future because it's all about people - it's about education and then about skills.
"And if you want to have skilled people, you'd better educate them properly before you start making them skilled."
Lord Heseltine, whose leadership challenge helped trigger Margaret Thatcher's departure from Downing Street in 1990, said standards need to be set by a strategic commission, chaired by the prime minister "that are relevant to the best in the world and not the 29th".
"If I could design an industrial strategy it would start in the primary schools," he said, adding that about a quarter of children leaving primary schools are "illiterate and innumerate" by modern employment standards.
"My own view is clear - there needs to be a much bigger devolution to the people who know where those schools are and they know the people that run them and they know where the inadequate results are coming from - in my personal experience of life: show me the problem, show me the person in charge."
Lord Heseltine cited one success story - Northamptonshire's "Race to the top" initiative - which aims to make Northamptonshire one of the highest performing counties for education by 2020.
"We spend a lot of time discussing the theories of education and the structures of education, but what we want is good head teachers," he said.
"Wherever you go in this country there is a skills shortage - that is a serious problem facing the expansion of the economy and it's a highly centralised process - not devolved on the scale that it would seem to me reflect the market opportunities."
Lord Heseltine says the UK's skills shortage is a serious problem - and if it was up to him he would start industrial strategy in primary schools.