A group chaired by Paralympic multi-gold medallist Baroness Tanni Grey-Thompson says it would be the first country in the world to take that step.
It would mean more time for sport, but unions worry too many core curriculum subjects could dilute their importance.
The Welsh government said it would consider the recommendation.
Education Minister Leighton Andrews said he wanted to look into how schools can increase levels of physical activity in children and young people.
By Arwyn JonesBBC Wales education correspondent
Although the need for more exercise is clear enough, it isn't just a simple matter of adding PE as a core subject.
Pupils in Wales already have as core subjects Welsh, English, maths and sciences.
They are the areas the government has decided should have most time and resources allocated to them.
And teaching unions fear the more you add to that core, the less of a priority they each become.
More than one-third are either overweight or obese, costing the Welsh NHS more than £70m a year.
The current core subjects in schools in Wales are English, Welsh (as a first language), mathematics, and science, while physical education is a foundation subject.
Having PE elevated to a core part of the curriculum would mean more time devoted to it, as well as extra resources and more training for teachers.
Schools would also need to give greater priority to provision for PE.
As a result:
• Teachers would be required to be competent to deliver the subject on completion of their initial teacher training.
• Well qualified, specialist teachers would be needed to work with schools to ensure high-quality provision.
• Standards and provision in physical education would be rigorously reviewed in school, both by the educational consortia and by schools inspectorate Estyn.
• Progress would be regularly measured.
The move to give PE more status in schools would be a bold move but an important one, according to the group.
In its report, the group, which also included the chair of Sports Wales Laura McAllister, said: "Given the Welsh government's commitment to making physical literacy as important a development skill as reading and writing, the group felt that changing the status of physical education is the only credible and secure way of ensuring this."
But while teaching unions support the idea, they are worried that too many subjects on the core curriculum could dilute their importance.
Dr Philip Dixon, director of ATL Cymru, said it was important to decide on "what small core we want".
"We certainly want literacy and numeracy in there," he said.
"We may well want physical education in there too, that's for the government to decide.
"But the core has to be kept very small or else we'll see a core which isn't a core and that's just ridiculous."
Owen Hathway, Wales policy officer at the NUT, also warned that schools already had a heavy curriculum in Wales.
"We are constantly being told that literacy and numeracy is a priority, " he said.
"Science is being pushed as a priority with its importance to future PISA testing. We have the core subjects of maths, English, Welsh and so on already in place, while IT is also being promoted as a priority.
"Quite simply, by the nature of the word, not everything can be a priority."
Responding to the report, Education Minister Leighton Andrews said he would be considering the report in the context of the wider review of the national curriculum and assessment.
"We firmly believe in the positive impact that physical activity can have on the people and nation of Wales and I'm determined to ensure that sport continues to make an important contribution to Welsh life," he added.
Sport Minister John Griffiths said sport was important as it improves health and wellbeing, along with helping with team building and making friends.
"We are committed to widening access and encouraging greater participation in physical activity as the health benefits of making physical exercise a part of everyone's daily lives is well-understood," he added.
PE should be given the same status as maths, English, science and Welsh in schools to help tackle obesity in Wales, experts have recommended.