David Niven ( March 1 , 1910 â `` July 29 , 1983 ) was an English actor . James David Graham Niven ( 1 March 1910 -- 29 July 1983 ) , known as David Niven , was a British actor and novelist , best known for his roles as Phileas Fogg in Around the World in 80 Days and Sir Charles Lytton , a.k.a. `` the Phantom , '' in The Pink Panther .