Karlovic, 35, served 21 aces as the world number one struggled to control his ground strokes in high winds.
In the semi-finals, Karlovic will face Spain's David Ferrer, who overcame Germany's Dustin Brown 6-2 6-2.
Third seed Tomas Berdych from the Czech Republic will face Italy's Andreas Seppi in the other semi.
Berdych saw off France's Richard Gasquet 6-2 6-1, while Seppi enjoyed a 7-5 6-4 victory over Ivan Dodig of Croatia.
But the day belonged to world ranked 27 Karlovic, who recorded his second career win over the Serb having also triumphed at the Madrid Masters in 2008.
Djokovic was nevertheless twice within two points of winning the match in the second set tie-breaker, before dropping serve in the seventh game of the final set.
Karlovic said: "I was feeling good despite the wind. I know he wasn't able to show his full range.
"I was relaxed and didn't feel I had to do anything extraordinary. I just had to focus on my game."
Novak Djokovic suffered a surprise loss against Croatia's Ivo Karlovic 6-7 (2-7) 7-6 (8-6) 6-4 in the Qatar Open quarter-finals in Doha.