The artwork, by US sculptor Charles Ray, will be replaced by a lamp-post at the tip of the Punta della Dogana.
The decision has angered some art lovers who argue that it shows a failure to embrace challenging, contemporary work.
But city officials say the sculpture was always intended to be temporary.
Boy with Frog, which is gleaming white and 8ft (2.5m) tall, has occupied the tip of a triangular piece of land that separates the Grand and the Giudecca canals for almost four years.
It shows a boy holding up a frog by one leg and gazing at it.
The sculptor had earlier expressed hope that his work would become a permanent "citizen of Venice".
"I never saw it as temporary," Mr Ray told the New York Times.
But some residents reportedly missed a 19th Century lamp-post, long a romantic meeting spot, which stood at the spot before.
Workers began to remove the sculpture overnight on Tuesday ready to clear the space early on Wednesday, Corriere del Veneto newspaper reported.
In its place will be a reproduction of the original lamp-post.
The old-fashioned landmark was thought to be more in keeping with the surrounding architecture, BBC Rome correspondent Alan Johnston reports.
Some animal-lovers also argued that the image of a boy holding the frog upside down was cruel, our correspondent adds.
But a piece in La Stampa newspaper criticised Venice's council, saying "administrative cowardice" had won out over real culture.
Defending its decision, the council said the sculpture had been considered temporary and Venice had a fine record of welcoming and exhibiting modern art.
Work has begun to take away a controversial statue depicting a naked boy with a frog in a part of ancient Venice, Italian media report.