As Mayo produced woeful shooting and Derry keeper Ben McKinless made several great saves, the visitors led by two with two minutes of normal time left.
But a Conor Loftus goal enabled Mayo to lead by three only for Mark Lynch's fisted goal to force extra-time.
However, Jason Doherty's goal helped Mayo take control early in extra time.
James Kielt had a penalty chance, which if converted, would have cut Mayo's lead but home keeper David Clarke was able to make a comfortable save.
Mayo's Patrick Durcan was sent off for an apparent off-the-ball strike late in the first period of extra time but they were still seven clear heading into the final 10 minutes of action and Derry were unable to summon up further resolve.
After a rather insipid Ulster SFC display against Tyrone, Derry regrouped to beat Waterford two weeks ago and their commitment in Castlebar was total.
The Oak Leafers looked the likely winners for most of normal time as Mayo's old failing of missed chances looked set to finally signal the end for this band of western warriors.
Substitute Loftus' last-gasp 1-1 appeared to have snatched victory for Mayo in normal time only for Lynch's goal to level matters and Kielt then had a difficult free chance to seal a Derry triumph only for his effort to miss the target.
Once Derry got into their stride, they set about cancelling out the early lead Mayo had established through Cillian O'Connor and Andy Moran.
Derry grew in confidence as the game progressed, taking on their opponents with pace on the flanks and quick movement of the ball, while keeping things tight at the back.
Niall Loughlin got them moving with a couple of scores, and it was Benny Heron who charged through the middle to fire over an inspirational lead point in the 16th minute.
Skipper Enda Lynn provided the pass for Ryan Bell to shoot them two ahead, and the home side went 10 minutes without a score, before O'Connor converted his second free.
Mayo almost had a goal when McKinless pushed a Moran effort on to a post, and denied O'Connor from the rebound, but Derry were also denied by goalkeeper David Clarke, who denied Carlus McWilliams, before the ball cannoned off a defender and on to a post.
The Ulster men lost midfielder Conor McAtamney to a black card just before the break, but thanks to a fourth score from Loughlin, they turned around with a narrow 0-8 to 0-7 lead.
Derry defended heroically through the second half with McKinless pulling off two more tremendous saves.
Kielt had just eased them two ahead when Mayo got the goal they so desperately needed, substitute Loftus smashing the ball to the net with just two minutes to play.
But the Oak Leafers refused to surrender, and pulled back the equalising goal through substitute Mark Lynch to tit it up at 1-12 each and send the tie to extra-time.
Another substitute Jason Doherty banged in a second Mayo goal, and then it was the turn of their goalkeeper David Clarke to perform heroics with a penalty save from Kielt.
Despite the loss of Patrick Durcan to a straight red card, they turned around with a seven points advantage, 2-17 to 1-13, O'Connor having brought his tally to ten points, including a first from play.
And they pushed the gap out to 11 points with points from O'Connor, who finished with 12, Doherty and Diarmuid O'Connor.
Derry's Chrissy McKaigue: "It's been a very difficult year. We have a very small squad because of a number of players are unavailable for various reasons.
"All you can ask if that everybody empties the task and I think we did that today. On another day, we must have snuck it.
"Defeat is always supposed to be devastating and it was definitely devastating today."
Saturday's GAA results
Derry ran out of steam in extra time in the qualifier against Mayo as last year's beaten All-Ireland finalists survived a nervy Castlebar test.