An HM Inspectorate of Prisons report into the Verne IRC in Portland also criticised the amount of legal representation available to detainees.
The report praised treatment of detainees by staff and the use of body cameras by managers.
The Home Office said it took detainees' welfare "extremely seriously".
HMP Verne was reclassified as an immigration removal centre in September 2014.
Almost 600 men awaiting deportation were held at the centre, with a majority having previously been in prison following criminal convictions.
During what was the first announced inspection since the change in status, inspectors found "levels of violence were too high and some of the violence was serious".
They reported a "concerted indiscipline" in the week before their visit in March.
They also pointed to higher than usual availability of new psychoactive substances and illicit alcohol.
The inspectors found 30 minutes of free legal advice was available and many detainees "struggled to obtain representation to fight their cases".
Their report also highlighted "excessive stays", including almost 40 men being held for more than a year, and one held for five years, which inspectors described as "one of the worse cases of prolonged detention we have seen".
It also said the former Victorian fort complex remained "too prison-like in character", with too much inner fencing and razor wire and restrictions on movement.
Chief Inspector of Prisons Nick Hardwick said: "Overall, The Verne was operating satisfactorily. However, despite considerable efforts to prepare the institution for its new role, the environment and staff culture reflected an institution that had not yet come to terms with its new function as an IRC."
A Home Office spokesperson said: "We are pleased the inspector has reported that the majority of detainees in The Verne felt safe and were very positive about their treatment by staff.
"We have considered the inspector's recommendations carefully and are committed to working with our contractors to implement our service improvement plan for the centre."
Former prisons ombudsman Stephen Shaw is to complete an independent review of detainees' welfare expected to be completed in the autumn.
High levels of violence were recorded at an immigration removal centre (IRC) in a former Dorset jail, the prison watchdog has found.