The dog's owners witnessed the incident at Bonna Point, a popular off-leash beach, said Sutherland Shire Council.
"We were pretty traumatised, it was horrific," one owner, named only as Nigel, told the Sydney Morning Herald.
Authorities have installed warning signs since the attack on Sunday and advised beachgoers and pet owners to avoid the water.
"New South Wales Department of Primary Industries advises people to avoid swimming or surfing when it is dark or during twilight hours," the council said in a statement.
The American Staffordshire Terrier, named Molly, was fetching a stick at the time and did not resurface, the owners said.
"It was very quick. It just took her under the water," Nigel was quoted as saying.
The incident was reported to involve a bull shark, but authorities could not confirm the species to the BBC.
The incident follows a recent series of shark sightings at nearby beaches this month.
On 9 February, a kite surfer was recorded on video riding towards a shark at Brighton Le Sands.
A few days later, a shark was spotted 10km (6 miles) inland at Alexandra Canal, near Sydney Airport, while another shark was seen in shallow water in Botany Bay.
Two people were killed by sharks in Australia last year - both in Western Australia - according to Sydney's Taronga Zoo.
Swimmers have been warned to avoid a Sydney beach after a pet dog was taken by a shark in shallow water.