The Pentagon chief also said Pyongyang would be "grossly overmatched" in a war against the US and its allies.
His stark warning comes a day after President Donald Trump threatened North Korea with "fire and fury".
US Secretary of State Rex Tillerson sought to assure Americans there was no imminent threat.
In Wednesday's strongly worded statement, Mr Mattis called on Pyongyang to halt its arms programme.
"The DPRK (Democratic People's Republic of Korea) must choose to stop isolating itself and stand down its pursuit of nuclear weapons," the US Marine Corps general said.
"While our state department is making every effort to resolve this global threat through diplomatic means, it must be noted that the combined allied militaries now possess the most precise, rehearsed and robust defensive and offensive capabilities on Earth."
Pyongyang's actions "will continue to be grossly overmatched by ours and would lose any arms race or conflict it initiates", Mr Mattis added.
Mr Trump - who has said he will not allow Pyongyang to develop a nuclear weapon capable of hitting the US - boasted earlier on Wednesday of America's atomic arsenal.
In early morning tweets from New Jersey where he is on holiday, the president said the US nuclear stockpile was "more powerful than ever before".
He added that he was hopeful "we will never have to use this power".
End of Twitter post  by @realDonaldTrump
End of Twitter post 2 by @realDonaldTrump
The tweets come amid escalating rhetoric between the two sides.
North Korea's state-run KCNA news agency said on Wednesday that Pyongyang was "carefully examining" a plan to strike the US territory of Guam.
The Pacific island is home to US military bases, strategic bombers and about 163,000 people.
Stopping in Guam to refuel after a trip to South East Asia, Mr Tillerson said he was hopeful a global "pressure campaign" involving Russia and China could lead to new dialogue with Pyongyang "about a different future".
America's top diplomat also defended President Trump's language.
He said North Korean leader Kim Jong-un did not understand diplomatic language, and a strong message was needed that he could understand.
Mr Tillerson added the situation had not dramatically changed over the last few days, and that Americans "should sleep well at night".
China has urged calm, describing the situation as "complex and sensitive".
Anthony Zurcher, BBC senior North America reporter
As Donald Trump made his "fire and fury" threat against North Korea, the lone camera in the room zoomed in for a tight shot. It would have been telling to see the effect those words had on his advisers.
Did they nod knowingly or look surprised? Was this part of the plan or an off-the-cuff presidential remark they'd have to clean up?
A day later and thousands of miles away, Rex Tillerson had out his mop and broom. The secretary of state said Mr Trump was merely reasserting that the US would defend itself if attacked, using language North Korea would understand.
That's not exactly what the president said, however. He warned of retribution if Kim Jong-un continued to make threats - punishing words with action.
It could simply be unfortunate phrasing from a man who is sometimes loose with words. Mr Tillerson would not be the first presidential aide intent on framing Mr Trump's blunt statements in a softer light.
With the president touting US nuclear might and test-firing an ICBM, however, the secretary of state - already considered to have little sway in this administration - may find his efforts at clarification buried in a torrent of bluster.
The White House on Wednesday denied reports that Mr Trump's "fire and fury" statement a day earlier, unusually provocative for a US president, was an off-the-cuff remark that had caught his inner circle unawares.
Press Secretary Sarah Huckabee Sanders told reporters: "The tone and strength of the message were discussed beforehand."
Despite rounds of UN sanctions, Pyongyang carried out two nuclear bomb tests last year and two intercontinental ballistic missile tests in July.
On Tuesday, media reports in the US claimed the North had achieved its goal of making a nuclear warhead small enough to fit inside its missiles.
However, most analysts doubt the tiny dictatorship would launch a suicidal pre-emptive attack on the US.
North Korea remains technically in a state of war with the US and South Korea after the 1950-53 conflict on the peninsula ended in a truce.
Get news from the BBC in your inbox, each weekday morning
US Defence Secretary Jim Mattis has told North Korea to avoid actions that could usher in the "end of its regime and the destruction of its people".