He was pulled from the water by the RNLI's Penlee Inshore Lifeboat at about 13:35 BST on Sunday, Devon and Cornwall Police said.
A crew member swam to the man and "recovered him" after he was found near the bow of an angling boat moored alongside Albert Pier, the RNLI said.
The man, said to be local, was airlifted to the Royal Cornwall Hospital in Truro and pronounced dead.
Police said the man had been reported floating face-down in the water by a sailing club.
A man died despite a rescue attempt at Penzance Harbour.