Hull keeper Eldin Jakupovic saved two penalties after the hosts had played the whole of extra time with 10 men following Dieumerci Mbokani's dismissal.
Newcastle had taken the lead in extra time when Mohamed Diame prodded home Vurnon Anita's cross to score against his former club.
The lead lasted a minute, Robert Snodgrass equalising after Matz Sels had spilled Markus Henriksen's shot into his path.
Mbokani was sent off in the 89th minute for lunging forward with his head at Jamaal Lascelles.
The tie only came to life in extra time, the first 90 minutes featuring just two shots on target.
Newcastle have now lost nine of their 10 penalty shootouts in all competitions, yet Rafael Benitez's side will be wondering why they did not win the tie in 120 minutes.
They had 32 shots in that time, but their finishing was as wayward as their penalties.
Jonjo Shelvey set the tone for Newcastle's spot-kick nightmare when his attempt was kept out by Jakupovic.
Hull's first penalty fell to Snodgrass and he made no mistake, before Dwight Gayle hit the bar with his attempt for Newcastle.
Skipper Michael Dawson made it 2-0 before Christian Atsu scored for Newcastle.
Tom Huddlestone made it 3-1, and Hull's players and fans were celebrating a first EFL Cup semi-final appearance when Jakupovic kept out Yoan Gouffran's tame attempt.
Hull had just one shot on target in normal time, and their hopes of reaching the semi-finals looked over when Mbokani was sent off.
The forward, on loan from Dynamo Kiev, kicked out and pushed Lascelles before lunging forward at the Newcastle defender with his head.
The 31-year-old has appeared in five league games since arriving at the end of August.
He now faces a three-match ban - something Hull could do without as they look to climb away from the Premier League relegation zone.
Hull City boss Mike Phelan: "This is the furthest the club has ever been and the players are really pleased.
"For this football club, it's a tremendous achievement. Hopefully we can go one step further now."
Newcastle manager Rafael Benitez: "The way that we played we deserved to score goals in the first half and in extra time. Then when we did score the goal we didn't manage the situation.
"It's something you learn with experience. Maybe we will do a little better the next time we have this many chances."
Newcastle have little time to dust themselves down. The Magpies return to Championship action on Friday away to Nottingham Forest (19:45 GMT). Middlesbrough await Hull in the Premier League at the Riverside next Monday (20:00 GMT).
Match ends, Hull City 1(3), Newcastle United 1(1).
Penalty Shootout ends, Hull City 1(3), Newcastle United 1(1).
Penalty saved! Yoan Gouffran (Newcastle United) fails to capitalise on this great opportunity,  right footed shot saved  in the bottom left corner.
Goal!  Hull City 1(3), Newcastle United 1(1). Tom Huddlestone (Hull City) converts the penalty with a right footed shot to the bottom left corner.
Goal!  Hull City 1(2), Newcastle United 1(1). Christian Atsu (Newcastle United) converts the penalty with a left footed shot to the high centre of the goal.
Goal!  Hull City 1(2), Newcastle United 1. Michael Dawson (Hull City) converts the penalty with a right footed shot to the bottom left corner.
Penalty missed! Still  Hull City 1(1), Newcastle United 1. Dwight Gayle (Newcastle United) hits the bar with a right footed shot.
Goal!  Hull City 1(1), Newcastle United 1. Robert Snodgrass (Hull City) converts the penalty with a left footed shot to the top right corner.
Penalty saved! Jonjo Shelvey (Newcastle United) fails to capitalise on this great opportunity,  right footed shot saved  in the bottom right corner.
Penalty Shootout begins Hull City 1, Newcastle United 1.
Second Half Extra Time ends, Hull City 1, Newcastle United 1.
Attempt missed. DeAndre Yedlin (Newcastle United) right footed shot from the right side of the box is too high.
Attempt blocked. Yoan Gouffran (Newcastle United) left footed shot from the centre of the box is blocked. Assisted by Jamaal Lascelles.
Attempt blocked. Mohamed Diamé (Newcastle United) right footed shot from outside the box is blocked.
Attempt missed. Dwight Gayle (Newcastle United) right footed shot from the centre of the box is too high. Assisted by Christian Atsu with a headed pass.
Attempt missed. Achraf Lazaar (Newcastle United) left footed shot from outside the box is just a bit too high. Assisted by Jonjo Shelvey following a corner.
Corner,  Newcastle United. Conceded by Michael Dawson.
Corner,  Newcastle United. Conceded by Josh Tymon.
Attempt missed. Markus Henriksen (Hull City) right footed shot from the left side of the box is just a bit too high. Assisted by Adama Diomande following a set piece situation.
Attempt missed. David Meyler (Hull City) right footed shot from the right side of the box misses to the left following a set piece situation.
Foul by Mohamed Diamé (Newcastle United).
Adama Diomande (Hull City) wins a free kick on the left wing.
Attempt missed. Dwight Gayle (Newcastle United) left footed shot from the centre of the box is high and wide to the left. Assisted by Christian Atsu.
Substitution, Newcastle United. DeAndre Yedlin replaces Vurnon Anita.
Corner,  Newcastle United. Conceded by Michael Dawson.
Attempt blocked. Yoan Gouffran (Newcastle United) left footed shot from the centre of the box is blocked.
Attempt blocked. Jamaal Lascelles (Newcastle United) header from the centre of the box is blocked. Assisted by Jonjo Shelvey with a cross.
Corner,  Newcastle United. Conceded by Josh Tymon.
Attempt saved. Dwight Gayle (Newcastle United) header from the centre of the box is saved in the top centre of the goal. Assisted by Christian Atsu with a cross.
Second Half Extra Time begins Hull City 1, Newcastle United 1.
Substitution, Newcastle United. Daryl Murphy replaces Isaac Hayden.
First Half Extra Time ends, Hull City 1, Newcastle United 1.
Attempt missed. Josh Tymon (Hull City) right footed shot from outside the box misses to the left. Assisted by Robert Snodgrass.
Foul by Ciaran Clark (Newcastle United).
Michael Dawson (Hull City) wins a free kick in the defensive half.
Attempt blocked. Yoan Gouffran (Newcastle United) left footed shot from the centre of the box is blocked.
Attempt blocked. Isaac Hayden (Newcastle United) right footed shot from outside the box is blocked.
Corner,  Newcastle United. Conceded by David Meyler.
Attempt missed. Jonjo Shelvey (Newcastle United) right footed shot from outside the box is high and wide to the right. Assisted by Mohamed Diamé.
Goal!  Hull City 1, Newcastle United 1. Robert Snodgrass (Hull City) left footed shot from the centre of the box to the centre of the goal.
Newcastle United missed three spot-kicks as Hull City beat the Championship leaders 3-1 on penalties to reach the EFL Cup semi-final.