Winterbourne View treats people with learning disabilities and autism.
Andrew McDonnell, who works with adults with mental disabilities, labelled some of the examples seen on film "torture".
All four have been released on police bail. The hospital's owners, Castlebeck, have apologised and suspended 13 employees.
Care Services Minister Paul Burstow said government regulator, the Care Quality Commission (CQC), had been asked to conduct an urgent investigation.
Avon and Somerset police confirmed that three men - aged 42, 30 and 25 - and a 24-year-old woman were arrested as part of their ongoing investigation into the hospital.
During five weeks spent filming undercover, Panorama's reporter captured footage of some of the hospital's most vulnerable patients being repeatedly pinned down, slapped, dragged into showers while fully clothed, taunted and teased.
The hospital is a privately owned, purpose-built, 24-bed facility and is taxpayer-funded.
NHS South West said it was "appalled" by the issues raised surrounding the care home.
In a statement, it said: "We always expect safe, high quality care from providers of services and the abuse of vulnerable patients is totally unacceptable."
Mr McDonnell, a clinical psychologist who viewed the footage, told the programme that basic techniques for dealing with patients with challenging behaviour were ignored.
He said he was shocked by some of the treatment of vulnerable patients.
After seeing footage of an 18-year-old patient named Simone being verbally abused and doused with cold water while fully clothed as a punishment, he said: "This is not a jail... people are not here to be punished.
"This is a therapeutic environment. Where's the therapy in any of this? I would argue this is torture."
Simone's parents told the programme that she had told them she was being abused at the hospital, but they had assured her that it would not be allowed to happen.
"She told us that she had been hit, her hair had been pulled and she'd been kicked - and I said no, this wouldn't happen, they're not allowed," said the patient's mother.
Professor Jim Mansell, from the University of Kent and a government adviser on the use of physical restraint for those with developmental disabilities, said that from the footage it appeared that staff were "waiting to pounce on people and restrain them".
"This is the worst kind of institutional care. It is the kind of thing that was prevalent at the end of the 60s and that led Britain to gradually close the large, long-stay institutions," he added.
The programme decided to film secretly after being approached by a former senior nurse at the hospital who was deeply concerned about the behaviour of some of the support workers caring for patients.
"I have seen a lot over 35 years but this I have never seen anything like this.  It is the worst I have seen," former nurse Terry Bryan told the programme.
"These are all people's sons, daughters, parents, aunties, uncles. These are all people who have got familiesâ€¦ the families themselves do not know what goes on there."
Mr Bryan reported his concerns to both management at Winterbourne View and to the CQC, but his complaint was not taken up.
Ian Biggs, regional director of the CQC for the southwest, said an opportunity to prevent abuse was missed when Mr Bryan's complaints were not investigated.
"Had we acted at that time, as we have done now, we can act very quickly to cease that kind of treatment.
"We missed that chance and we are sorry for that and we're doing everything we can now to make sure we're responding properly."
In a statement, the Care Quality Commission said, following an internal review, it recognised that "there were indications of problems at this hospital which should have led to us taking action sooner".
"We apologise to those who have been let down by our failure to act more swiftly to address the appalling treatment that people at this hospital were subjected to," it said.
Reporter 'haunted' by abuses
Mr Burstow said people deserve to receive "safe and effective care" from every care provider.
He also said he had "confirmed with CQC that they should undertake a series of unannounced inspections of services for people with learning disabilities."
Castlebeck has launched an internal investigation into their whistle-blower procedures and are reviewing the records of all 580 patients in 56 facilities.
The vulnerable patients filmed by Panorama have been moved to safety.
The hospital charges taxpayers an average of Â£3,500 per patient per week and Castlebeck has an annual turnover of Â£90m.
Chief executive Lee Reed told the programme he was "ashamed" by what had happened.
"All I can do is unreservedly apologise to both the families and the vulnerable adults that have been involved in this and recommit to making sure this doesn't happen again," he said.
Panorama reporter Joe Casey said he was shocked by what he witnessed.
"On a near-daily basis, I watched as some of the very people entrusted with the care of society's most vulnerable targeted patients - often, it seemed, for their own amusement. They are scenes of torment that are not easily forgotten," he said.
South Gloucestershire Council said it "takes all allegations of abuse and mistreatment of vulnerable adults very seriously".
In a statement, it said: "Our immediate concern is always for the safety and welfare of patients.
"As soon as the SAB [Safeguarding Adults Board] were made aware of these allegations at Winterbourne View... appropriate action was taken in line with established procedures and protocols."
Panorama's Undercover Care: The Abuse Exposed was broadcast on BBC One on Tuesday 31 May at 2100 BST and is available to view in the UK on the BBC iPlayer.
Police in Bristol have arrested four people after secret filming by BBC Panorama found a pattern of serious abuse at a residential hospital.