Moderate exercise increased the size of the hippocampus, an area of the brain that makes memories, in 120 volunteers.
The year-long trial, published in Proceedings of the National Academy of Sciences, showed performance on memory tests also improved.
Exercise may buffer against dementia as well as age-related memory loss.
The latest work looked at healthy people in their 60s rather than people with Alzheimer's or other forms of dementia.
But the findings have important implications for ageing societies faced with a dementia time bomb.
In the UK, 820,000 people have dementia, and this figure is set to double by 2030.
Until a cure is discovered, finding cheap and simple ways to reverse this trend is imperative, say experts.
Professor Kirk Erickson and colleagues from the University of Pittsburgh in the US set out to investigate the impact that even moderate exercise might have on preserving memory.
They split their 120 volunteers into two groups. One group was asked to begin an exercise regimen of walking around a track for 40 minutes a day, three days a week, while the others were limited to doing simple stretching and toning exercises.
Brain scans and memory tests were performed at the start of the study, halfway through the study and again at the end.
Scans revealed hippocampus volume increased by around 2% in people who did regular aerobic exercise. The same region of the brain decreased in volume by 1.4% in those who did stretching exercises, consistent with the decrease seen in normal ageing.
Both groups showed some improvement over time on memory tests. In the walking group, the improvement appeared to be linked with increased size of the hippocampus.
Professor Erickson said: "We think of the atrophy of the hippocampus in later life as almost inevitable. But we've shown that even moderate exercise for one year can increase the size of that structure.
"The brain at that stage remains modifiable."
Dr Simon Ridley of the Alzheimer's Research Trust said that although the study does not look at memory loss in Alzheimer's or dementia, it suggests "it's never too late to start exercising to help keep our brains healthy".
"Even modest exercise may improve memory and help protect the brain from normal decline caused by ageing.
"Increasing evidence suggests regular exercise and a healthy diet may help reduce our risk of developing dementia as well as reaping numerous other benefits from living a healthy lifestyle."
Walking for 40 minutes a few times a week is enough to preserve memory and keep ageing brains on top form, research shows.