Mayor Stephanie Rawlings-Blake said she was replacing Police Commissioner Anthony Batts with his deputy, Kevin Davis, for an interim period.
The city was rocked by riots in April when a black man died after suffering injuries in police custody.
Six officers were charged over the death of the 25-year-old, Freddie Gray.
Speaking at a news conference on Wednesday, Mayor Rawlings-Blake said Mr Batts had "served this city with distinction" since becoming police chief in October 2012.
But referring to the city's high homicide rate, she said "too many continue to die".
"The focus has been too much on the leadership of the department and not enough on the crime fighting," she told reporters, adding: "We need to get the crime surge under control."
The city has seen a sharp increase in violence since Freddie Gray's death on 19 April, with 155 homicides this year, a 48% increase over the same period last year.
On Tuesday, the police department announced that an outside organisation will review its response to the civil unrest that followed Mr Gray's death.
The US justice department is also conducting a civil rights review of the Baltimore force and Mr Batts has been criticised by the city's police union.
Earlier on Wednesday, the union released its report into the police handling of the rioting.
It said officers had complained "that they lacked basic riot equipment, training, and, as events unfolded, direction from leadership".
The report also said "officers repeatedly expressed concern that the passive response to the civil unrest had allowed the disorder to grow into full scale rioting".
Recent events had "placed attention on police leadership", Ms Rawlings-Blake said, but denied her decision was influenced by the union report.
Mr Davis, who is taking over immediately as interim police chief, praised his "friend" Mr Batts and said he was a "true reform commissioner".
Mayor Rawlings-Blake said Mr Davis would "bring accountability to police, hold officers who act out of line accountable for their actions".
Baltimore's mayor has sacked the US city's police chief, saying his leadership had become a distraction from fighting a "crime surge".