Locals fear it will damage the environment and pollute their water sources.
La Colosa, in Central Colombia, has the potential to become South America's largest gold mine.
Mr Arce said the town's decision could not be applied retroactively.
The minister added that the South African mining giant AngloGold Ashanti had already been issued an exploration licence, which would retain its validity.
Mr Arce also said that while the land was under the control of local authorities, any subterranean riches were under the control of the national government.
The minister said that if AngloGold Ashanti was awarded the environmental licence it needs to proceed with the project, the courts or Colombia's congress would have to decide whether the local or national authorities prevailed.
Mr Arce insisted that the referendum vote would not affect foreign investment in Colombia's mining sector.
"For the first time in 20 years we will have three major projects in execution phase," he told RCN.
He said campaigners had misinformed voters.
"We're not talking about an open-air mine here," he said. "Nor are there a hundred rivers at risk."
Campaigners were angered by Mr Arce's comments.
The opposition to President Juan Manuel Santos also criticised the government's stance on the issue.
"Santos and the Mining Minister agreed to disregard Cajamarca's decision on La Colosa. Democratic mobilisation will also defeat them," tweeted opposition Senator and presidential hopeful Jorge Robledo.
Only 76 residents of Cajamarca, in the central province of Tolima, voted in favour of the mining project in Sunday's referendum, while more than 6,100 of the town's 19,000 inhabitants voted against.
Several other Colombian cities and towns are planning to hold similar votes to try to block mining projects in their area.
Correction 29 March 2017: This story has been amended to clarify the government's position on the mining ban.
Colombia's Mining Minister German Arce has questioned whether the result of a referendum held in the town of Cajamarca, where 98% of residents voted against a major gold mining project, will prevail.