The Commons Public Accounts Committee said whistleblowers had often been subjected to bullying and harassment.
Its report called for whistleblowers to be offered legal and counselling help and for "swift sanctions" to be imposed on staff who victimised them.
The government said it was acting to ensure people felt free to speak out.
The report highlighted the "important" role whistleblowers had played in uncovering details about the Hillsborough disaster and the Mid Staffordshire NHS trust scandal.
The committee said whistleblowing was a "crucial source of intelligence to help government identify wrongdoing".
However, it found there had been a "startling disconnect" between policies encouraging whistleblowers in theory and what happened in practice.
The report said it had "heard of too many cases of appalling treatment of whistleblowers by their colleagues".
Officials who tried to raise concerns often had to show "remarkable courage" in coming forward.
The report highlighted whistleblowing in relation to the Hillsborough disaster on 15 April 1989, where 96 Liverpool fans at an FA Cup semi-final lost their lives.
Prime Minister David Cameron later apologised for the "double injustice" of the disaster, after an independent report published in 2012 showed police and emergency services had attempted to deflect the blame.
It also highlighted the Mid Staffordshire NHS scandal, where staff at Stafford Hospital had blown the whistle on "appalling care" between 2005 and 2008.
The committee heard from Kay Sheldon, a member of the board of the Care Quality Commission who, the report said, had been "victimised" by senior officials after she tried to raise concerns about the way it had been operating.
The report said no-one had faced any form of sanction over her treatment.
Labour MP Margaret Hodge, who chairs the public accounts committee, said whistleblowing was "crucial" and must be taken seriously by all chief executives of major companies and public sector organisations.
She said protection for whistleblowers was still not adequate, citing the example of Osita Mba - a former lawyer at Revenue and Customs who drew attention to what he claimed were "sweetheart deals" between the tax authorities and Goldman Sachs.
She said Mr Mba was a "really brave guy" but it had become "impossible for him" to remain with the organisation after his disclosures and he had had "to start his life again elsewhere".
"The way you support whistleblowers is hugely important," she added.
"I think it is really important that there are proper sanctions in place in an organisation so if someone does blow the whistle, they are properly supported and if anyone dares bully or harass them, they are not only reprimanded but punished."
Legal protections for whistleblowers
The 1998 Public Disclosure Act protects workers who disclose information about potential criminal behaviour and other malpractice at their workplace, or former workplace, provided certain conditions are met.
Depending on the nature of information disclosed and who it is confided to, a whistleblower is legally protected from suffering from any detriment as a result. If these conditions are not met, a disclosure may constitute a breach of the worker's duty of confidence to his employer.
The laws apply to direct employees and agency workers but not those working for the security services, and campaign group Public Concern at Work says other individuals - such as volunteers and interns - are also excluded.
The law was amended last year to include a specific public interest test, whereby whistleblowers have to "reasonably believe" that their actions are in the public interest to be protected.
Workers who raise concerns about bullying and harassment by work colleagues are also protected. Colleagues who victimise them are personally liable for their actions while their employers are  "vicariously liable".
The Taxpayers' Alliance said the report suggested there was a "culture of secrecy" in much of the public sector and whistleblowing must be encouraged.
"Though the practice often asks tough questions, it is a crucial tool to increase the accountability of those in charge and to ensure our public services work as well as possible for the people who need them," the campaign group's chief executive Jonathan Isaby said.
A government spokeswoman said: "All civil servants must be able to raise concerns so that poor services and inefficient operations can be identified and acted upon.
"That is why this government is ensuring people feel free to speak out. Awareness of how to raise a complaint has risen by 20% and two-thirds of civil servants feel that any complaint will be investigated properly.
"In addition, we are ensuring that all departments have a clear whistleblowing policy."
Labour said whistleblowing had led, in many cases, to major changes in policy.
"There must be a clear standardised policy for handling whistleblowers working across all of our public services to ensure that there are no disparities in how people are dealt with, and that staff are protected from being victimised." said shadow Cabinet Office minister Michael Dugher.
The treatment of employees in public services who have raised concerns about wrongdoing has often been "shocking", a group of MPs has said.