Known as "The Bulldozer", he won with 58% of the vote to the 40% of his main rival Edward Lowassa.
The victory margin was a surprise - many analysts had predicted a tighter race after four opposition parties united behind Mr Lowassa's candidature.
It boiled down to Mr Magufuli's popularity, which superseded that of the governing Chama Cha Mapinduzi (CCM) that has dominated politics since independence from British rule in 1961.
So what are the five big challenges he faces once in office?
Change was the buzzword during the election campaign. Now people will expect him to engage in a major house-cleaning exercise by showing the door to corrupt government officials and old-school public servants.
This will pose a huge political risk, as they are well-entrenched in CCM. But Mr Magufuli, the former works minister, will have to tackle corruption head-on, as Tanzanians have had enough of it.
His first big test will come with the appointment of a cabinet.
Will it be made up of people untainted by corruption? If not, voters will be disappointed.
After all, he has built his political career on the promise that he is not scared of taking risks, and that he works hard and acts fast.
Although the official unemployment rate is at around 10%, people sometimes get the impression that the crisis bigger - a recent advert for 70 jobs in the public service posts attracted more than 10,000 applications. Interviews had to be held in a football stadium.
Almost half of Tanzania's 50 million-strong population is made up of young people, and they are worst-affected by unemployment.
They voted with great enthusiasm, believing their fortunes will change under Mr Magufuli.
He will be under most pressure from them to deliver on his promise to create jobs - and end poverty.
He has promised to revive industries which are in bad shape, like the cotton and fish processing sectors.
Mr Magufuli has promised children free education from kindergarten to secondary school.
While many Tanzanians welcome this, they feel it is not enough. They want a higher standard of education with improved curriculums so that Tanzania can have a better-skilled work force.
Mr Magufuli will be hard-pressed to meet their demands - almost all state schools are short of desks, books and teachers.
With last week's local elections in Zanzibar scrapped because of alleged rigging, Mr Magufuli will have to make sure that a re-run takes place, and that it is free and fair.
When Mr Kikwete came to power, he promised to keep a close eye on the semi-autonomous islands, which are popular with tourists.
But as he leaves the office, more than 60% of Zanzibaris want more autonomy for the archipelago - an indication of their dissatisfaction with the union.
If Mr Magufuli can bring about a lasting solution, he will go down in history as one of Tanzania's most successful presidents - but it is a big if.
Outgoing President Jakaya Kikwete promised a new constitution, but failed to deliver on it during his 10 years in office.
The opposition gained many votes with its promise to make sweeping changes to the constitution - including reducing presidential powers and guaranteeing the impartiality of state institutions, like the electoral commission.
Mr Magufuli ignored the issue during the election campaign. Can he afford to do that throughout his presidency? Unlikely.
Tanzania's President-elect John Magufuli will be inaugurated on Thursday following his landslide victory in fiercely contested elections.