The Division One leaders started day four on 48-3, chasing 235 for victory.
Steve Eskinazi (30) nicked Jake Ball (4-54) behind and Nick Compton was run out for 63 to leave the visitors 146-5.
But 58 not out from John Simpson and captain James Franklin's unbeaten 54 saw them to a victory which left them one point clear of Yorkshire.
Defeat for bottom side Notts, who have won only once in the championship this season, means they will be playing Division Two cricket in 2017 for the first time in a decade.
Victory for Middlesex maintained their unbeaten record in four-day cricket in 2016, although second-placed Yorkshire's 23-point win over Durham reduced Middlesex's lead at the top by three points.
Somerset, winners inside three days over Warwickshire this week, are 22 points behind Middlesex in third.
Middlesex play Lancashire at Old Trafford next week, while Yorkshire entertain Somerset at Headingley.
Middlesex meet Yorkshire at Lord's in a potential title decider starting on Tuesday, 20 September.
Despite Middlesex's modest target in Nottingham, Ball's breakthrough and Compton's run-out after a mix-up on the stroke of lunch left the game in the balance.
But the composed Simpson, who brought up his sixth half-century of the summer, and 35-year-old Franklin sealed victory with an unbroken stand of 89.
Compton, who took a two-month break from cricket this summer, told BBC Radio London: "I was proud of the way I managed to hold things together.
"As someone who prides myself on tough situations, I wanted to make sure I was on it.
"The intent was to make sure we found a way to do it, because this is the time.
"You don't get too many opportunities to be a county player playing for the title as this stage of the season. It is a fantastic place to be."
Notts director of cricket Mick Newell, who will hand first-team coaching duties to former England coach Peter Moores at the end of the season, said: "It's about the embarrassment of getting relegated.
"Financially it doesn't make a huge difference. It is more about the impact it has on the ego of the players and the impact on the supporters who are used to seeing Division One cricket.
"We are proud club; we have been in Division One for a long time and it's 10 years since I stood here and had to explain away a relegation."
Middlesex maintained their bid for a first County Championship title since 1993 and relegated Nottinghamshire with a five-wicket win at Trent Bridge.