The Spaniard pounced to drive low into the corner from Zlatan Ibrahimovic's pull-back as United set up a home tie with West Ham.
Paul Pogba had hit a post moments earlier as the game came alive in the second period, following a first half in which neither of the bitter rivals managed a shot on target.
City boss Pep Guardiola made nine changes to his side, who are now six games without a win in all competitions - his worst run as a manager.
Their best chance arrived early on when Kelechi Iheanacho headed over from six yards, while second-half substitutes Raheem Sterling and Sergio Aguero made little impact.
Victory was United's second in five matches, and only Jose Mourinho's fourth win in 18 meetings with Guardiola.
Former Chelsea and Real Madrid manager Mourinho said on Wednesday his life in Manchester is "a bit of a disaster".
His words and United's poor form created a sense that defeat at Old Trafford would prompt tough headlines, and Mourinho admitted recent displays influenced his choice to make only four changes and retain key names.
United, far from at their best, scored during their only spell of quality in the opening 10 minutes of the second half as  Ibrahimovic - anonymous in the opening period - profited from Nicolas Otamendi's missed header to square for Mata.
There remain areas of concern for United. Luke Shaw was constantly outwitted by Jesus Navas and again there was no place in the 18-man squad for £26m summer signing Henrik Mkhitaryan.
But while Guardiola said the competition was his "lowest priority" before kick-off, Mourinho perhaps took the chance to shift United's momentum.
Guardiola said City must be "more clinical" after they ended the night without working David de Gea once - the first time they have not had a shot on target in a game since 2012.
City, who are top of the Premier League, had an early penalty appeal turned down when Michael Carrick tangled with Aleix Garcia.
The wholesale changes made for the tie may blur detailed analysis of just what is amiss in recent weeks, but a switch to a 4-4-2 formation after two league games where a back three was used perhaps shows Guardiola has not yet settled on the way forward at City.
He could also face more time without captain Vincent Kompany, who, making only his second start of the season, asked to be replaced at half-time because he was tired.
Kompany suffered a groin strain in September but has been dogged by thigh and calf issues, prompting his playing time to fall from around over 2,700 minutes in 2013-14 to 1,700 last season.
There were some positives for Guardiola as 19-year-old full-back Pablo Maffeo made made some key interventions but if City fail to win on Saturday they would be seven matches without a win, a dire run not endured since 2004.
Manchester United manager Jose Mourinho on responding to Sunday's 4-0 league defeat at Chelsea: "We were on a good run of results but it was a big defeat - numbers that the history of this club doesn't deserve.
"When I'm in a club my heart belongs to the fans and I felt deeply for them.
"The reality is that I never had people like these ones. We lost 4-0 and they were supportive.
"Today the stadium was full of real support and it looks like the love people have for the club is bigger than bad results, bigger than three bad seasons. We must give something back."
Manchester City boss Pep Guardiola: "I'm proud of the young players and how they played.
"At that level you need to be more clinical with the last pass but it's difficult against United. We are happy. I want to try to win but now we'll focus on the next competitions."
Manchester City travel to the side who have made fewer passes than anyone in the Premier League this season - West Brom - on Saturday and host Barcelona three days later.
United face Burnley at Old Trafford on Saturday and visit Fenerbahce in the Europa League on Thursday.
Match ends, Manchester United 1, Manchester City 0.
Second Half ends, Manchester United 1, Manchester City 0.
Foul by Pablo Maffeo (Manchester City).
Zlatan Ibrahimovic (Manchester United) wins a free kick in the defensive half.
Zlatan Ibrahimovic (Manchester United) is shown the yellow card.
Paul Pogba (Manchester United) is shown the yellow card for a bad foul.
Nicolás Otamendi (Manchester City) wins a free kick in the defensive half.
Foul by Paul Pogba (Manchester United).
Foul by Aleksandar Kolarov (Manchester City).
Zlatan Ibrahimovic (Manchester United) wins a free kick in the attacking half.
Offside, Manchester United. Paul Pogba tries a through ball, but Morgan Schneiderlin is caught offside.
Foul by Sergio Agüero (Manchester City).
Michael Carrick (Manchester United) wins a free kick in the defensive half.
Substitution, Manchester United. Jesse Lingard replaces Marcus Rashford because of an injury.
Attempt blocked. Jesús Navas (Manchester City) right footed shot from the left side of the box is blocked. Assisted by Aleksandar Kolarov.
Sergio Agüero (Manchester City) wins a free kick in the defensive half.
Foul by Luke Shaw (Manchester United).
Corner,  Manchester United. Conceded by Pablo Maffeo.
Foul by Aleix García (Manchester City).
Morgan Schneiderlin (Manchester United) wins a free kick in the defensive half.
Nicolás Otamendi (Manchester City) wins a free kick in the defensive half.
Foul by Morgan Schneiderlin (Manchester United).
Foul by Fernando (Manchester City).
Zlatan Ibrahimovic (Manchester United) wins a free kick on the right wing.
Substitution, Manchester United. Morgan Schneiderlin replaces Juan Mata.
Gaël Clichy (Manchester City) wins a free kick in the defensive half.
Foul by Marcus Rashford (Manchester United).
Corner,  Manchester City. Conceded by Daley Blind.
Substitution, Manchester City. Sergio Agüero replaces Nolito.
Attempt missed. Aleksandar Kolarov (Manchester City) left footed shot from outside the box is high and wide to the right from a direct free kick.
Raheem Sterling (Manchester City) wins a free kick in the attacking half.
Foul by Ander Herrera (Manchester United).
Corner,  Manchester City. Conceded by Luke Shaw.
Ander Herrera (Manchester United) wins a free kick in the defensive half.
Foul by Raheem Sterling (Manchester City).
Substitution, Manchester City. Raheem Sterling replaces Leroy Sané.
Attempt missed. Aleksandar Kolarov (Manchester City) left footed shot from outside the box is too high. Assisted by Aleix García.
Offside, Manchester United. Antonio Valencia tries a through ball, but Juan Mata is caught offside.
Goal!  Manchester United 1, Manchester City 0. Juan Mata (Manchester United) left footed shot from the centre of the box to the bottom left corner. Assisted by Zlatan Ibrahimovic.
Attempt missed. Ander Herrera (Manchester United) right footed shot from outside the box misses to the right.
Juan Mata struck to win a tight Manchester derby as United knocked out EFL Cup holders City to reach the quarter-finals.