Rescuers earlier ended efforts to find the body of Jeffrey Bush, 36, whose bedroom was engulfed by the 6m (20ft) hole on as he slept on Thursday night.
Sinkholes are common in parts of Florida, although most are quite small.
Experts are trying to ascertain how far the underground cavity reaches and whether more homes are at risk.
All homes near to the house in Seffner - a suburb of 8,000 people 25km (15 miles) east of central Tampa - have been evacuated.
Officials have created a 100ft-wide safety zone fearing the top of the sinkhole is growing.
The house covers most of the hole, and a local official said crews would try to move the building forward to enable surviving family members to retrieve their belongings from it.
"We don't know, in fact, whether it will collapse or whether it will hold up,'' said Hillsborough County Administrator Mike Merrill.
On Sunday, the demolition crews left standing only a few walls of the house.
They were proceeding as carefully as possible to salvage at least some of the home's contents for five survivors.
A Bible, family photos and a woman's purse were among the items saved.
The demolition is expected to be completed on Monday and experts would then examine the hole.
Just before the crews started their work, 35-year-old Jeremy Bush, who had tried to save his brother, was seen praying outside the house.
Five people escaped unharmed, after hearing a loud crash and Jeffrey Bush screaming as the earth crumbled below the house.
Jeremy Bush said he jumped into the chasm to dig his brother out but failed. He was later rescued by police.
Rescuers then used listening devices and cameras at the scene, but found no signs of life.
"There's no way of possible survival," said fire rescue spokeswoman Jessica Damico on Saturday.
Mr Bush told reporters that someone had visited the four-bedroom house some weeks ago to check for sinkholes and other issues but found nothing wrong.
"And a couple of months later, my brother dies. In a sinkhole," he said.
Florida state law requires home insurers to provide coverage against sinkholes.
Experts say much of Florida sits on a system of limestone caverns which are subject to water erosion, causing them to collapse.
Demolition crews have partially razed the home of a man presumed dead after he was swallowed up by a sinkhole in Florida.