The dairies operation, which processes and distributes milk, is being sold to Germany's Mueller for Â£80m.
It comes as profits at the UK's largest dairy food company fell 95% to Â£900,000 in the six months to September.
Dairy Crest processes and delivers around 1.3 billion litres of milk a year for retailers and homes.
Dairy Crest said in a statement that the deal was in the best interests of consumers, customers and dairy farmers.
The dairies business accounts for about 70% of the company's revenues, which rose 1% to Â£682.1m during the six months.
After the sale, which still needs shareholder approval and could take several months, Dairy Crest will focus on its profitable cheese and spreads operations.
There are about 14,000 dairy farmers in the UK, producing 3.3 million litres a day. However, with milk prices having fallen, there has been much debate about whether the economics of the industry are sustainable.
Investors approved of the Dairy Crest's decision to get out of a loss-making sector, sending its shares 10% higher in morning trading on Thursday.
Muller said the deal would lead to lower costs and larger exports of dairy products made in the UK.
Ronald Kers, chief executive of Muller UK & Ireland, said: "We are concerned that the dynamics of the UK fresh milk market are unsustainable for dairy processors in the mid to long term and this acquisition will allow us to reduce our costs, increase our efficiencies and invest in the future."
Under the deal, Mueller's UK division - Muller Wiseman Dairies - will take over factories at Foston, in Derbyshire, Chadwell Heath, in Essex, and Severnside, near Gloucester.
The deal also includes the Hanworth glass bottling site in Middlesex, where Dairy Crest is consulting with employees on the site's future, and 72 depots.
Muller bought Robert Wiseman in 2012.
Dairy Crest, maker of Cathedral City cheese and Country Life butter, has announced a big slump in profits and the sale of its milk business.