The Grade II-listed market cross in Ilchester was knocked down after being hit by a car on Friday.
A 28-year-old man was arrested by police in the early hours on suspicion of drink driving.
Town trustee Graham Mottram, said: "Our major challenge is to put the monument back together again using as much of the original material as we can."
The cross is a Grade II-listed building, which dates from about 1795.
Mr Mottram said they will need to find a stonemason and "some pretty big lumps of ham stone" to replace the column.
"The base has been dismantled by the impact and we're going to have a structural survey as to what needs to be done to stabilise the base," he said.
"I certainly expect it will cost us a five-figure sum, running into six."
Repair work to a Somerset monument destroyed in a crash will run into five figures, the town trust has said.