Groups of tourists, fresh off the ferry from Cape Town, pass by with barely a glance.
But for Mr Mandela, the garden held a special value. And if you stand inside the tiny cell where South Africa's struggle icon spent 18 years hidden from the world, and peer through the barred window into the high-walled courtyard, you can just make out the thin strip of green and brown foliage where he once tended tomatoes and chillies.
"A garden was one of the few things in prison that one could controlâ€¦ The sense of being the custodian of this small patch of earth offered a small taste of freedom," Mr Mandela wrote in his autobiography, Long Walk To Freedom - an early draft of which, incidentally, he tried to hide from the prison guards by burying in the garden.
Today, the garden's weeds - and the rotting stump of an apple tree - hint at the alleged mismanagement and corruption that have blighted Robben Island for years.
But along one wall, two grape vines now stand out in sharp contrast - the lush green foliage of a promising harvest suspended on a wooden frame.
"They're still growing, thriving, regardless of the difficult conditions on the island. They're very much a symbol of the resilience of the leaders imprisoned here," said Philip Jonker, a South African wine maker as he and two colleagues carefully pruned the vines by hand.
It is seven years since Mr Jonker first spotted the two plants on a family visit to the island.
It is not clear whether Mr Mandela himself tended the vines or whether, as seems more likely, they were planted shortly after he was transferred to another prison on the mainland. But Mr Jonker sees them - there are seven vines around the prison in total - as part of the island's heritage, and a way of preserving "South African history in a bottle".
After four years of petitioning, he finally secured permission to tend the vines, and, last year, to take the first harvest - previous ones had been devoured by the island's ravenous birds - back to his Weltevrede family wine estate two hours' drive east of Cape Town.
At this moment - leaving aside the current headlines about low pay and grim conditions in the local wine industry  - you may be wondering if this is simply another potentially uncomfortable attempt to exploit the Mandela "brand".
The short answer is no. Although Weltevrede will surely not mind the publicity.
"Mandela is my grape father," said Weltevrede worker Jan Blaauw, with a broad grin, as he and Mr Jonker pulled away some dead branches.
The staff make regular trips to the island - their progress with the vines offering an increasingly sharp contrast with the visible neglect seen elsewhere here.
Mr Mandela "brought us everythingâ€¦ Freedom. This is my way to pay him back for all he has done for me," said Mr Blaauw.
Thirty workers at Weltevrede have been given their own small vineyard to farm as part of an empowerment scheme designed to help address the economic legacies of apartheid.
Any profits from the auction of the wine will be split between the workers' trust, another local charity, and an organisation helping disadvantaged families of former Robben Island political prisoners.
Not that significant profits are assured. The harvest is small, to put it mildly.
Back at the family estate, in a stunning valley surrounded by mountains, Mr Jonker sniffed and then sipped a sample of the Methode Cap Classique sparkling white wine that is still fermenting.
Sixteen small bottles of sweet white dessert wine have already been produced and the aim is to fill 94 magnums with the sparkling wine.
It is still not clear exactly which grape variety is growing on Robben Island, but Mr Jonker believes the quality will be high, and is now trying to research the world of charity auctions.
The workers at the farm appear to share Mr Jonker's enthusiasm.
"It can benefit our children. It's a great experience because I can go out and say I own vines and I can make wine, and we also make wine for Mandela. He showed us how to look forward and not to look back," said Maggie Kawula, standing beside a row of Pinot Noir vines owned by the workers' trust.
The magnums won't be ready for auction for at least a year or so. But having sipped a tiny mouthful of the half-fermented vintage - which will be called The Manuscript, in reference to the story about Mr Mandela burying his autobiography in the prison garden - I can tell you it tastes just fine.
"In another 50 or maybe 100 years' time those vines may still be alive and those may be the only living monuments of the times of our great leaders on the island," said Mr Jonker.
In the sun-bleached corner of Nelson Mandela's old prison yard on Robben Island, the desiccated remains of a tiny, weed-ridden garden lie exposed and sadly neglected.