Police said the elderly man was spotted in the water at Leith at about 22:00 on Wednesday.
He was taken to Edinburgh Royal Infirmary by ambulance for treatment, but later died.
A Police Scotland spokesman said inquiries into the circumstances surrounding this incident were ongoing.
A man has died after being pulled from the water near the Royal Yacht Britannia at Ocean Terminal in Edinburgh.