It comes in a shake-up of UK military buildings and resettling of regiments.
Brecon and Radnorshire Conservative MP Chris Davies condemned the closure, saying there had been a barracks in Brecon since 1805, home to troops who fought the Zulus at Rorke's Drift.
"This decision is abhorrent and I shall be fighting it every step of the way," he said.
"The government has a great deal of questions to answer over why it is proposing to close a well-loved and historic barracks in a vitally important military town.
"Brecon Barracks has served our country with distinction over its long history, with soldiers from the site fighting in every conflict since the early 19th century.
"This decision shows a blatant lack of respect for that history."
Mr Davies said he was launching a petition against the decision, saying the Brecon area had some of the highest unemployment levels in Wales.
He also hoped the closure would not damage the town's "thriving" military tourism industry.
Brecon barracks has about 85 civilian staff and 90 military but it is not thought jobs are at risk.
Mr Davies said he understood the nearby Sennybridge training ground and infantry school at Dering Lines would not be affected.
Defence Secretary Sir Michael Fallon told the Commons on Monday the reorganisation in Wales would see a specialist light infantry centre created at St Athan, Vale of Glamorgan.
Cawdor Barracks, Pembrokeshire - whose closure was previously announced in 2013 - will now shut in 2024, while a storage depot at Sennybridge will go in 2025.
Responding for Labour, Shadow Defence Secretary Nia Griffith, MP for Llanelli, said the ministry was "right to restructure its estate".
But she warned closing bases would affect the livelihoods of many people who would face "gnawing uncertainty" over their future.
The Ministry of Defence is to close the British Army's Welsh headquarters at Brecon, Powys, in 2027.