Hanako, which means "Flower Child", was 69 and the oldest elephant in Japan.
The elephant was a gift from Thailand when it was two years old.
Last year a petition was launched to improve the animal's conditions, but Hanako was deemed too old to move back to Thailand.
A spokesman for Inokashira Park Zoo on the outskirts of Tokyo said Hanako was found lying on her side on Thursday morning and efforts to raise her were not successful. She died in the afternoon.
The war on elephants
Have we fallen out of love with elephant rides?
Hanako was a popular attraction in the zoo, but the elephant, born in the wild, spent most of her life caged and alone in a concrete enclosure with little greenery.
Last year a Canadian blogger wrote an article about the elephant's enclosure, describing it as a "concrete prison".
That prompted the petition, which led the zoo to provide new toys and make other changes.
However, the zoo said she was too old to be moved and an independent US wildlife expert said staying in a sanctuary with other elephants would bewilder Hanako after living so many years alone.
An elephant that was the focus of an international campaign to improve its living conditions has died in a zoo in Japan.