The Court of Cassation rejected Mr Sarkozy's challenge to the seizure.
The diaries were initially confiscated as part of an inquiry into alleged illegal funding during his successful 2007 presidential campaign.
That case has since been dropped, but the diaries may be used in other investigations targeting Mr Sarkozy.
The former president, who lost his re-election bid in 2012, is planning a political comeback and correspondents say the drip of allegations has harmed him.
The diaries were seized after claims surfaced in 2010 that Mr Sarkozy had taken advantage of 90-year-old L'Oreal heiress Liliane Bettencourt when he was standing for president.
It was alleged that Mrs Bettencourt had given large sums of cash to Mr Sarkozy's aides. Both Mr Sarkozy and Mrs Bettencourt denied this. Last October, Mr Sarkozy was removed from the list of defendants.
Lawyers for Mr Sarkozy argued that confiscating the diaries had been illegal.
But in Tuesday's ruling, the Court of Cassation - France's top court - decided there was no need to rule on the issue as Mr Sarkozy was no longer a suspect in the case.
The ruling comes days after Le Monde newspaper reported that Mr Sarkozy had recently had his phone tapped on orders from judges investigating alleged campaign donations from late Libyan ruler Muammar Gaddafi.
The papers said the phone taps had revealed evidence of tampering with the justice system. Mr Sarkozy denies the claims.
French media say the diaries could be used in this case, and also in an investigation into allegations that French tycoon Bernard Tapie received a huge payout in 2007 to settle a long-running legal battle with the French state.
In a separate development regarded as embarrassing for the former president, transcripts of recorded conservations taped by a former aide of Mr Sarkozy were leaked last week.
Mr Sarkozy and his wife Carla Bruni say they were recorded without their consent.
A French court has ruled investigators can retain the seized diaries of ex-President Nicolas Sarkozy, who is battling corruption allegations.