Media playback is not supported on this device
The 34-year-old, who won 10,000m gold eight days ago, was swamped by his rivals in the final lap and Ethiopia's Muktar Edris broke clear to win gold.
Britain's Farah kicked again to take silver at the London Stadium in his final major track championships.
Britain's men won a stunning 4x100m relay gold as the women took silver.
"It's been a long journey but it's been incredible," Farah told BBC Sport.
"It didn't quite hit home until after I crossed the line and had a couple of moments to myself when I realised - this is it.
"I gave it all - I didn't have a single bit left at the end."
Four-time Olympic champion Farah will finish his track career with a record of 10 golds and two silvers in major championships.
He is set to focus on marathons after his final track appearance at the 5,000m Diamond League final in Zurich on 24 August.
Gold and silver in the two 4x100m relays takes Britain's medal tally at London 2017 to four - Farah supplying the other two - but the hosts are still two short of UK Sport's target of six to eight.
Edris, 23, ran a superbly judged final 600m to claim his first major title with a sprint finish reminiscent of so many Farah triumphs.
After a brief burst from Farah's compatriot Andrew Butchart at the front of the race, the Ethiopian trio of Edris, Yomif Kejelcha and Selemon Barega moved up to box in the home favourite.
Barega fell back but Edris used Kejelcha as a pace-setter into the home straight before surging away while Farah battled for position with the resurgent Paul Chelimo behind.
Farah finally found open track but Edris was gone, with the Briton passing the fading Kejelcha and holding off Chelimo to salvage silver.
"The 10,000m took a lot more out of me than I realised," said Farah.
"I was trying to cover every move but they had a game plan - one of them was going to sacrifice himself and that's what they did and the better man won."
Farah has now lost just two of his 22 5,000m contests since 2011.
"I was highly prepared for this race and I knew I was going to beat Mo Farah," said Edris, who won in a time of 13 minutes 32.79 seconds to claim his first win over the Briton in five attempts.
"I am the new champion for Ethiopia, that's why I did the 'Mobot' [Farah's traditional celebration] - I am the next champion."
1976 Olympic 10,000m bronze medallist and BBC commentator Brendan Foster: "We did say fatigue was going to get him. It has to come to an end. He hates losing."
BBC athletics expert Allison Curbishley: "In my head and my heart I didn't think anyone could genuinely beat Mo Farah but that 10,000m at the start of the week took so much out of him. Edris, we take our hat off to you. This man has beaten the legend that is Mo Farah in his own back yard."
2005 marathon world champion Paul Radcliffe: "He just doesn't have the pace in his legs anymore to be able to live with challengers like that but this isn't the end of Mo Farah in athletics. He has a new challenge ahead of him in the marathon."
Two-time Olympic decathlon champion Daley Thompson: "He has got to be hurting. But it has to end for everybody. In a couple of weeks or months, he will look back and say, 'that was awesome'."
2004 Olympic 4x100m relay champion Darren Campbell: "There are athletes out there who have won more medals, but the quality of those medals and for Mo to have led an event on his own... wow. I don't know how you can't see him as our greatest ever athlete."
Media playback is not supported on this device
A disconsolate Farah dropped to the track as Edris mimicked his famous celebration, before finally picking himself up to complete a farewell lap.
Farah has come under scrutiny in recent years, with his coach Alberto Salazar under investigation by the US Anti-Doping Agency.
The Briton also recently denied any wrongdoing after computer hackers revealed some of his blood samples had initially been flagged as suspicious, before later being declared normal.
After initial shock at the result, the crowd found their voice to give Farah a final warm send-off at the stadium he lit up with double Olympic gold in 2012.
"Back in 2008 at the Olympics I was so disappointed [at not making the 5,000m final] and I didn't know what to do with myself, whether to carry on with the sport," said Farah.
"I had to decide what it took to become a champion and that was to move to the other side of the world - nothing was guaranteed but I took that chance.
"This London crowd is amazing - there is no place like home.
"I have a few more races on the track, then I will take a short break and then this chapter of my life is closed."
Elsewhere, Britons Katarina Johnson-Thompson and Morgan Lake finished fifth and sixth respectively in the high jump as Maria Lasitskene - the Russian competing as a neutral athlete - won gold with a leap of 2.03m.
And Ashley Bryant finished 11th in the men's decathlon, with Kevin Mayer becoming the first French world champion in the event.
Media playback is not supported on this device
Mo Farah missed out on a fifth major championships distance double in a row as he finished second in the 5,000m at the World Athletics Championships.