Eleven care workers are to be sentenced later for the maltreatment and neglect of five patients at Winterbourne View.
But a BBC Inside Out West investigation has found evidence of alleged abuse involving different staff members.
Police have agreed the assault on Ben Pullar was "not acceptable" but said they would not re-open the case.
Mr Pullar is autistic and bi-polar, with severe learning difficulties, and when he was 18, he spent almost a year at Winterbourne View from July 2009.
He lost two teeth when he was punched in the face.
His twin, Tom, said his behaviour changed drastically after going to the hospital.
"He's burst out crying on me and demanded to talk about Winterbourne View - and this is two years after Winterbourne View - so it still affects him today," said the 21-year-old.
The BBC has seen daily care notes from his time at the hospital which state Mr Pullar "had an accident with his teeth" just weeks after he arrived.
His family said they were given mixed messages about what happened and were told at one point he had "bitten the floor".
He was taken to Bristol Dental Hospital by Winterbourne View staff as an emergency patient, and two doctors there raised the alarm with the authorities because of the severity of his injuries.
Senior dental consultant Dr Jane Luker said: "I think they thought the injury he sustained wasn't consistent with what they were being told.
"He had allegedly bitten a carer and the injuries were sustained from him biting that carer.
"[But] because the teeth were pushed back palatally, towards the back of the mouth, it would be consistent with a punch or a blow to the front of the face."
The authorities met several days after Mr Pullar was assaulted, but his family was not invited.
Tom Pullar said: "I think that care staff are supposed to care for patients, not punch them."
Notes obtained by Inside Out said the nurse was bitten and retaliated in order to remove his fingers - and later, that he had pushed and pulled to get them out.
The nurse involved, Maxwell Nyamukapa, was suspended and later reinstated. Despite repeated attempts to contact Mr Nyamukapa, he has refused to comment.
The police logged it as an assault, and said the nurse had acted instinctively and in self-defence.
Forty concerns were raised about patients in just over three years with the local safeguarding board run by South Gloucestershire Council.
The police were also contacted 29 times, nine of those times related to carers suspected of using restraint of involved in violent incidents with patients - including Mr Pullar.
Det Ch Supt Louisa Rolfe, of Avon and Somerset Police, said: "We have carried out an extensive investigation into incidents at Winterbourne View and no stone has been left unturned.
"We consulted with the CPS and our partners in terms of action we could take.
"Looking back on that incident now and understanding the whole pattern of what was happening at that residential hospital, it is clearly not acceptable.
"However, at the time, the officer was dealing with information, reliable information, that came from patients and carers, and that came from other experts in that field. And perhaps they relied upon that information too much."
She added: "We accept that there is learning for us as an organisation and we will do things differently in future."
Det Ch Supt Rolfe said the force had carried out an extensive investigation into incidents at the hospital and had improved how it dealt with people with learning disabilities.
The council has declined to comment on Mr Pullar's case.
Castlebeck, the hospital's owner at the time, said its new board and management had already, and would continue to, rigorously review what happened at Winterbourne View.
The firm said if any other matters come to light, immediate action would be taken and the appropriate authorities informed.
It has also referred Mr Nyamukapa twice to the Nursing & Midwifery Council, who declined to comment.
Inside Out West is on BBC One West at 7.30pm on Monday or for seven days after on the iPlayer.
The family of a man who was punched in the face as a patient at a private hospital in Bristol is calling for the incident to be re-investigated.