The fighters are said to be "hiding in fewer than 10 houses" in the seafront district of Al-Giza al-Bahriya.
The pro-unity government force announced its full control of Sirte on Monday, after the last IS-controlled area fell to the UN-backed fighters.
However, a spokesman has told the BBC they are not declaring victory yet.
The operation's Ahmed Hadia told the BBC: "They've now taken control of the last area, but that does not mean military operations have ended."
Mr Hadia said the forces, which have been supported by US air raids, would now focus on "combing the city" to find any remaining IS jihadists.
Sirte was IS's stronghold following its capture in June 2015, and had repeatedly portrayed it in propaganda as its main base outside of Syria and Iraq.
But the group was forced back into just one small neighbourhood in late October by pro-government forces following a seven-month battle for control.
According to the Facebook page of the force backing the Government of National Accord (GNA), IS had been using women and children as human shields during the battle.
Pictures of their apparent captives emerging from the area, covered in dust, have been shown on Libyan television.
"Today they [forces loosely allied to GNA] helped many women and over 20 children, mostly infants and toddlers, who came out terrified and in dire need of medical and post-traumatic assistance," the Facebook page said.
It has been a long and deadly battle for the forces who fought to oust so-called Islamic State from Sirte.
The end of the battle there will be welcomed by thousands who had to flee their homes, and by the international community.
Although this does not spell the end of the radical group's presence in the country, it does significantly weaken it. It is likely that its members will increasingly resort to staging more deadly, isolated attacks. It still has cells operating under its banner in different parts of Libya.
In 2015, when IS was driven out from the first Libyan city it ever controlled - Derna - it reverted back to control under a mix of local Islamist militias with links to al-Qaeda.
Libya's complex network of local armed groups, and its continued dysfunctional politics, still provide ripe territory in which extremist groups can thrive.
In the months ahead, Libya's vast array of politicians and rival administrations will need to start finding tangible solutions to the ongoing civil conflict, which led to Sirte's dramatic fall into IS hands in the first place.
Libyan forces are hunting down the last Islamic State (IS) jihadists in the city of Sirte, after ousting them from their former bastion.