A report following an unannounced inspection said the overall number of self-harm incidents at HMP Woodhill was also "very high".
It said the prison's response to the deaths had "lacked rigour".
The National Offender Management Service said a new governor was "working hard" to improve things.
The high-security jail, which is mainly used to hold remand, newly-sentenced and short-term prisoners, was inspected in January.
The report said recorded levels of assaults were "very high and double what inspectors have seen in other local prisons".
But it said the "general environment in the prison was good" and the accommodation was "reasonable".
Nick Hardwick, chief inspector of prisons, said: "Woodhill is a prison that retains great potential, particularly in terms of the resources at its disposal.
"However, as we reported last time, outcomes for prisoners remained concerning and some were very disappointing."
Michael Spurr, chief executive of the National Offender Management Service, said the governor and his staff had a "unique set of challenges".
"Woodhill holds some of the most difficult and disruptive men in the prison estate," he said.
"The governor and his staff are working hard to reduce the incidents of violence and to improve rehabilitation."
Inspectors have raised concerns about the number of suicides at a Buckinghamshire prison, after four "self-inflicted deaths" last year.