But the committee said imposing financial financial penalties on those who did not meet targets would be counter productive.
Scotland is committed to reducing emissions by at least 80% by 2050.
But a recent report confirmed the country again missed its target for emissions in 2012.
It was the third time an annual target had not been met.
The Scottish Parliament's Rural Affairs, Climate Change and Environment Committee voiced its support for mandatory reporting as a government consultation on climate change closes at the end of this month.
The committee heard evidence from public bodies and found that "everyone stressed how seriously they take their responsibilities both to reduce their carbon footprint, and to report on their climate change obligations".
It said there were different methods of reporting between local councils, voluntary organisations, police and other emergency services.
And it said clarification was needed before mandatory reporting was introduced.
Committee convener Rob Gibson, an SNP MSP, said: "The legislation passed by the Scottish Parliament commits Scotland to reducing emissions by at least 80% by 2050 and is world-leading in terms of ambition.
"However that ambition has to be translated into action. It is now time to activate real change in approach and behaviour across all underperforming sectors of society, and this includes the public sector.
"The committee is encouraged to hear many examples of the positive impact voluntary reporting has had on actually reducing emissions, and by the unanimous support for mandatory climate reporting by the public sector bodies who gave evidence.
"We must support those who are not delivering as much as they can to make Scotland a world leader, and encourage them to engage and improve."
Net emissions in 2012 were 55.67 million metric tons of carbon dioxide equivalent (MtCO2e), compared to the target of 53.226.
Welcoming the committee's recommendation, WWF Scotland director Lang Banks said: "If Scotland is to get itself back on track to meeting its climate change objectives then its important everyone plays their part, and that includes Scotland's public sector bodies.
"With better coordination and support from central government, public sector bodies could be slashing their climate pollution while also saving money through initiatives to improve energy efficiency or by generating their own renewable energy."
Public sector bodies in Scotland should be compelled to report climate emissions targets and progress, a Holyrood committee has recommended.