Prisoner distress, self-harm and suicide attempts are too often seen as signs of manipulation, rather than vulnerability, research by the Centre for Mental Health and the Howard League for Penal Reform found.
A record number of prisoners killed themselves in 2016, figures have shown.
The government said an internal inquiry into deaths in custody was under way.
The report by the two charities, which is based on interviews with healthcare staff working in prisons, found that the majority of prisoners have multiple and complex needs, including poor mental health.
But many do not get access to mental health support.
The report claims that staff shortages, a "toxic" working environment and a failure to recognise the impact of prison on both prisoners and staff, all contribute to an "unsafe environment".
It recommends three steps to help prisons shift away from a "primarily punitive approach" to a culture centred on wellbeing, recovery and rehabilitation:
Sarah Hughes, chief executive of the Centre for Mental Health, said: "We need to bring about a culture change in prisons that puts safety top of the agenda, that understands the traumas both prisoners and staff too often live with, and that means people get the right help when they need it."
Last month, the Ministry of Justice said there were 119 self-inflicted deaths in prisons in England and Wales in 2016 - 29 more than the previous year and the highest number since records began in 1978.
Frances Crook, chief executive of the Howard League for Penal Reform, said it was "time for action".
She added: "No-one should be so desperate while in the care of the state that they take their own life.
"Staff who work in prisons should never feel so under pressure that they cannot stop and listen."
A government spokeswoman said: "Increased support is now available to those at risk of self-harm or suicide, especially in the first 24 hours, and we have invested in mental health awareness training for staff.
"An internal inquiry is also under way looking at all deaths in custody in the last year, to further our understanding of why these events happen."
Prisons in England and Wales need a "profound culture change" which focuses on inmate safety, a new report says.