Gray opened the scoring in the 52nd minute with a fine solo goal before Slimani doubled the lead shortly after.
The Algerian scored his second four minutes later as he bundled home Gray's cross from close range.
Caolan Lavery's fortuitous late strike cut the deficit but Musa fired home in stoppage time.
The first half was largely uneventful as both sides made eight changes from the weekend's league matches.
Gray's goal was the first moment of quality in the match as he collected the ball 30-yards from goal on the left wing, drove in field and beat three defenders before slotting through Blades' goalkeeper Jake Eastwood's legs.
Slimani then scored a four-minute brace which looked to have settled the tie as the striker - signed for £29m in 2016 - showed excellent movement and composure.
But, Lavery pulled a goal back in the 83rd minute as his miscued cross beat Ben Hamer at the near post before the Blades pushed further with 19-year-old Foxes debutant Josh Knight making a fine block to deny Billy Sharp.
The Leicester defence continued to hold strong and in the 93rd minute Musa powered through the United defence and drove home to confirm the result.
Match ends, Sheffield United 1, Leicester City 4.
Second Half ends, Sheffield United 1, Leicester City 4.
Goal!  Sheffield United 1, Leicester City 4. Ahmed Musa (Leicester City) right footed shot from the centre of the box to the top right corner.
Attempt blocked. Ahmed Musa (Leicester City) right footed shot from the centre of the box is blocked.
Islam Slimani (Leicester City) wins a free kick in the attacking half.
Foul by David Brooks (Sheffield United).
Foul by Ahmed Musa (Leicester City).
David Brooks (Sheffield United) wins a free kick on the right wing.
Attempt missed. David Brooks (Sheffield United) header from the centre of the box is too high. Assisted by John Lundstram with a cross following a corner.
Corner,  Sheffield United. Conceded by Josh Knight.
Attempt blocked. Billy Sharp (Sheffield United) right footed shot from the centre of the box is blocked. Assisted by Caolan Lavery.
Corner,  Leicester City. Conceded by Chris Basham.
Attempt missed. Islam Slimani (Leicester City) right footed shot from the centre of the box misses to the left. Assisted by Ahmed Musa.
Goal!  Sheffield United 1, Leicester City 3. Caolan Lavery (Sheffield United) right footed shot from a difficult angle and long range on the right to the bottom right corner. Assisted by David Brooks.
Foul by Nampalys Mendy (Leicester City).
Caolan Lavery (Sheffield United) wins a free kick in the defensive half.
Substitution, Leicester City. Josh Knight replaces Harry Maguire.
Corner,  Leicester City. Conceded by Chris Basham.
Attempt blocked. Demarai Gray (Leicester City) left footed shot from the left side of the box is blocked. Assisted by Ahmed Musa.
Substitution, Leicester City. Nampalys Mendy replaces Wilfred Ndidi.
Foul by Wilfred Ndidi (Leicester City).
Samir Carruthers (Sheffield United) wins a free kick in the attacking half.
Video Review:.
Offside, Leicester City. Ahmed Musa tries a through ball, but Wilfred Ndidi is caught offside.
Corner,  Leicester City. Conceded by John Lundstram.
Attempt blocked. Ben Chilwell (Leicester City) left footed shot from the left side of the box is blocked. Assisted by Ahmed Musa.
Corner,  Sheffield United. Conceded by Ben Chilwell.
Substitution, Sheffield United. Paul Coutts replaces George Baldock.
Substitution, Sheffield United. Chris Basham replaces Jake M Wright.
Goal!  Sheffield United 0, Leicester City 3. Islam Slimani (Leicester City) right footed shot from very close range to the bottom left corner. Assisted by Demarai Gray with a cross.
Corner,  Leicester City. Conceded by Jake Eastwood.
Attempt saved. Demarai Gray (Leicester City) right footed shot from outside the box is saved in the top right corner. Assisted by Andy King.
Attempt blocked. Wilfred Ndidi (Leicester City) right footed shot from outside the box is blocked.
Corner,  Sheffield United. Conceded by Wilfred Ndidi.
Goal!  Sheffield United 0, Leicester City 2. Islam Slimani (Leicester City) left footed shot from the left side of the six yard box to the bottom left corner. Assisted by Andy King with a through ball.
Substitution, Sheffield United. Billy Sharp replaces Ched Evans.
Christian Fuchs (Leicester City) wins a free kick in the defensive half.
Foul by George Baldock (Sheffield United).
Offside, Sheffield United. Ched Evans tries a through ball, but George Baldock is caught offside.
Offside, Leicester City. Daniel Amartey tries a through ball, but Islam Slimani is caught offside.
Goals from Demarai Gray, Islam Slimani (2) and Ahmed Musa took Leicester past Championship side Sheffield United and into the EFL Cup third round.