Why does A* fail to find the fastest path when it reaches the goal?
Your misunderstanding is that A* doesn't return success until it pops a goal node, in your terminology.

So, in your example, when it pops&nbsp;$C$, it puts $D$&nbsp;in the open list, with quite a high value. It then pops&nbsp;$B$ and puts $D$ in the open list again, with a lower value. So the first time that $D$ is popped will be from the route $A\to B\to D$, as it should be.