DR Congo could only draw 1-1 with their rivals in Kinshasa, having drawn the first leg of their qualifier 0-0.
It put Congo Brazzaville through on the away-goals ruling.
In other shocks South Africa, Mali and Algeria were also eliminated.
DR Congo boasted an outstanding record in the previous four tournaments for home-based footballers with two titles and two quarter-finals finishes.
They were expected to beat Congo Brazzaville in front of their home fans in Kinshasa and qualify for the 2018 Championship in Kenya next January and February.
Jean-Marc Makusu Mundele put the title-holders ahead on 36 minutes only for Jaures Ngombe to equalise just two minutes later.
DR Congo needed at least one more goal to survive, but Congo defended bravely to clinch only a second appearance at the 16-team tournament.
Severely depleted South Africa contained Zambia for 79 minutes in Ndola before conceding two goals in three minutes to lose the match 2-0 and the tie 4-2 on aggregate.
Justin Shonga bagged a brace for Zambia against a South Africa side deprived of many first choice players because clubs refused to release them.
Mali, who lost the 2016 final against DR Congo in Kigali, were another shock casualty as they had a man sent off when losing 1-0 at home to Mauritania.
Karamoko Traore scored on 38 minutes in Bamako and the hosts were reduced to 10 men before half-time when Boubacar Samassekou was shown the red card with Mauritania advancing 3-2 on aggregate.
Nigeria secured a ticket to Kenya by overcoming Benin 2-0 in Kano. Trailing 1-0 from the first leg, the Nigerians secured their place through goals from Rabiu Ali and Kingsley Eduwo.
Ivory Coast, third behind DR Congo and Mali at the 2016 Nations Championship, scored seven minutes into stoppage time to pip Niger for a place at next year's finals.
Sanfo Sylla scored the crucial goal to give the Ivorians a 1-0 victory and an away goals victory after the tie ended 2-2.
Angola missed a penalty in Luanda before Dany scored the only goal of the tie just past the hour to inflict a first loss in 11 matches on Madagascar.
Rwanda staged a brave comeback to defeat Uganda 2-0 in Kigali but failed 3-2 on aggregate, with Cameroon and Sudan also booking places in Kenya.
On Friday, Morocco and Libya knocked out Egypt and Algeria respectively as they booked their places at Kenya 2018.
Morocco beat Egypt 3-1 in Rabat on Friday to win the tie 4-2 overall.
Libya held Algeria to a 1-1 draw in Sfax to go through to the finals with a 3-2 aggregate victory.
Despite CHAN matches only featuring locally-based footballers the games have full international status and count toward the monthly Fifa rankings.
The African Nations Championship (CHAN) holders, the Democratic Republic of Congo, failed to qualify for the 2018 finals in Kenya following a shock elimination by Congo Brazzaville.