Mark Drakeford has already apologised for the scandal on the Tawel Fan ward at Glan Clwyd Hospital, Bodelwyddan.
It followed a damning report into what went on with families saying patients were treated like animals in a zoo.
But he said the embarrassment to the Welsh government was the "least important part" of it all.
"The real focus is to make sure the families and the people who are closely connected to what went on is that we properly attend to what they need," Mr Drakeford told BBC Wales.
He added he is yet to meet any of them but he was "hugely impressed" by the way the families have acted and they are "looking for assurances" that there will not be a repeat of what went on.
"I'm very willing to speak to the families when the time is right," he said. "This is a really tough time for these families."
'It was probably a mistake keeping him in there'
Mr Drakeford said on Thursday he will consider putting Betsi Cadwaladr health board into special measures which could mean the Welsh government or another body taking over some or all of the board's functions.
In an interview with BBC Radio Wales on Friday, Mr Drakeford said he will receive advice via a "proper process" involving the Welsh government, Health Inspectorate Wales and the auditor general for Wales which will meet to review whether further intervention is needed.
"It isn't a matter of a politician shooting from the hip, making an off the cuff decision," he said.
"We get the experts involved together and we act on the proper advice that we are given."
Meanwhile, the boss of patient watchdog North Wales Community Health Council (CHC) said the health board needed a "greater level of external support and direction".
Chief officer Geoff Ryall-Harvey said: "The avalanche of adverse reports over the past few months has been unprecedented in my experience of 30 years as a CHC chief officer."
The families of patients who suffered "institutional abuse" at a mental health ward in Denbighshire will have a chance to meet the health minister.