Simons blamed construction costs and archaeology requirements for its decision to ask West Dorset District Council to find alternative developers.
West Dorset District Council had agreed funding of up to Â£4m towards the scheme which includes Marks & Spencer and Waitrose stores.
The council said it was "disappointed" by the decision.
The company cited the requirements of English Heritage, including undertaking a full archaeological survey, as making the project financially unviable.
In a letter to the council, Simons' chief executive Tom Robinson said the viability of the second phase of the project had been "marginal for a number of years".
An online petition set up by residents' group Defend our Rich Cultural Heritage backing calls for archaeological investigations of the site, which lies within the walls of the old Roman town, has received almost 4,000 signatures.
Anthony Alford, leader of West Dorset District Council, said: "We remain ambitious for this key town centre site.
"Our vision is to keep Dorchester a thriving shopping destination and keep retailers in our county town."
New council buildings, a library and an adult learning centre have already been built in Charles Street in the first phase of its regeneration.
The developer of a new retail area in Dorchester town centre has pulled out of the scheme.