Mr Justice Openshaw told the Old Bailey the 87-year-old peer had "advanced and disabling dementia".
Both prosecution and defence barristers had agreed at a previous hearing that Greville Janner was not well enough to take part in a criminal trial.
Lord Janner is accused of 22 counts of sex offences against boys.
The peer, who was suspended from the Labour Party in April, and his family have always denied any wrongdoing on his part.
He is accused of 15 counts of indecent assault and seven counts of a separate sexual offence against a total of nine alleged victims.
The Old Bailey heard the allegations spanned the years 1963 to 1998. Twenty-one of the charges relate to children who were aged 16 or under at the time.
The judge, who made his decisions based on a series of medical reports from four experts, said: "The defendant has advanced and disabling dementia that has deteriorated and is irreversible, and accordingly I find that he is unfit to plead."
The judge's ruling was made on the basis that Lord Janner would not be able to understand the charges against him, instruct lawyers or enter a plea.
Mr Justice Openshaw summarised the reports, saying that while the public had been "unjustifiably sceptical" over the case, they had a "right to know" how he had come to his decision.
Lord Janner is unable to remember key events from his life, with one report from February 2014 saying he could not name his children or grandchildren, remember where he had gone to school, or recall details of his parliamentary career.
The most recent report, from September, stated that Lord Janner has no short-term memory, barely recognises his family and "can now hardly speak".
A "trial of the facts" is scheduled take place next April, when a jury will decide if he committed the abuse, with no finding of guilt or conviction.
It is a relatively rare procedure normally used in cases where a defendant poses a danger and needs to be hospitalised for the safety of themselves and others.
Director of Public Prosecutions Alison Saunders decided in April that it was not in the public interest for Lord Janner to face a criminal trial because of his dementia.
Her decision was based on expert reports, two of which were ordered by the defence and two by the prosecution.
It was overturned following an independent review requested by six of the alleged victims and the decision on Lord Janner's fitness to plead was then handed over to the High Court judge who has now made a ruling on the matter.
Greville Janner was made Lord Janner of Braunstone in 1997. He remains a life peer, but has been on leave of absence since 2014.
Lord Janner was not required to attend court on Monday. A further hearing was set for 7 March.
Lord Janner is unfit to stand trial over allegations of child sexual abuse spanning four decades, a High Court judge has ruled.