But since the downturn started, the town has faded to be a shadow of its former self.
"Unfortunately, Lithgow survives on the mines, which are closing down," she told the ABC.
"There's not as many people hanging around, retail is dropping off."
Ms Matthews said up to seven local businesses have closed in the past 18 months.
"It was a very, very thriving town," she said.
"We need to try and get people wanting to live in Lithgow, rather than just the retiring tree-changers."
Down the street at the bakery, locals come and go, but there aren't many young faces.
Owner David Dowler fears for the future of his town.
"I hope it doesn't turn into a ghost town, but you know, it might," he said.
"We need more jobs for young people, you have got to keep the young people around."
Some blame Lithgow's problems on job losses at the Wallerawang Power Station, which shut in 2014.
The state-owned power station was sold to power company Energy Australia, but soon after the sale it was mothballed and has sat idle since.
Lithgow's Deputy Mayor, Wayne McAndrew, said nearby mines started laying off workers around the same time.
"We're going through a difficult time, and anyone who lives here would know that," he said.
"We had the closure of the Wallerawang Power Station back in 2014, and very quickly after that, the closure of the Angus Place colliery.
"That was a loss of over 600 permanent, full-time jobs, and you can understand the flow-on effect of that," he said.
There has been a push to give the site of the Wallerawang Power Station a new life, redeveloping it to bring new jobs to the region.
"I'm resigned to the fact, sadly, that we'll never again see power generated at Wallerawang," Mr McAndrew said.
"But we're being told the site is very valuable for future job opportunities, if it isn't demolished completely."
Energy Australia has announced recycling company Bettergrow will be granted exclusive access to the site so it can decide whether it wants to purchase it.
"We were looking at part of the site for recycling and resource recovery operations," Bettergrow General Manager Neil Schembri said.
"Since then the opportunity and the site has evolved, we're now looking at purchasing the whole site."
Mr Schembri said once approvals were in place he would like to see up to 30 jobs created over a two-year period.
"It's realistic to expect in years to come there'd be 200-300 people in jobs on the site there," he said.
While re-development is the talk of the town in Lithgow, some are concerned about who could end up paying for any demolition works at the power station site.
In a recent media release, Energy Australia said it had applied for permission to the Lithgow City Council, "to rehabilitate plant infrastructure and surrounding land".
"This involves … potentially removing the primary boiler, turbine hall, cooling towers, buildings and associated equipment such as coal-handling conveyors, above-ground pipelines and electrical systems," it said.
Lithgow Deputy Mayor Wayne McAndrew said taxpayers could be picking up the bill for some of the works, because of an agreement between the State Government and Energy Australia.
"We're told quite clearly that if the site is to be levelled, the cost of that is to be borne by the residents and taxpayers of NSW, and the taxpayers of NSW," he said.
But the State Government said it cannot reveal if that is the case.
"NSW Treasury is working with Energy Australia on the decommissioning, demolition and rehabilitation of the Wallerawang site," it said in a statement.
But it said the details of the sale agreement were confidential.