Speed of electric signal vs chemical diffusion
Membrane potential is a directed force, while diffusion is a random walk

Your confusion is entirely justified, as you're correct that both diffusion and depolarization rely on the speed of ion travel. Where they differ is that depolarization is a directed force, occuring only in one direction along the membrane, whereas diffusion is subject to the stochastic walk of ions in water.

When membranes depolarize, the whole membrane begins with a certain amount of "pressure" built up behind it. As the signal propagates, a small area of this pressure is released at a time and triggers the next section to trip. That next section is essentially identical to the previous one, with the same amount of pressure behind it initially. In this way, you can imagine membranes "conserving" the initial concentration, so that the signal strength isn't dependent on the distance from the signal itself.

Diffusion, on the other hand, is stronger initially and decays over distance because of its stochastic nature. It's a probabilistic force, and so depends in general on something like the square root of the distance. If you imagine a thousand ions taking a step forward in one moment, in the next it's equally likely that five hundred take a step backward even as an equal amount moves forward. So the signal degrades over time and distance from the source.

If that was confusing, the following metaphor may help. Depolarization acts like a line of dominos, while diffusion acts like a complex dance. Check out this video of the largest human mattress dominos: the speed they're falling over is pretty constant when they're not turning. Compare that to two dancing humans who are equally likely to move forward and backward, and hopefully you can see that the mattress dominos is the faster way to go, even though each one is theoretically limited by how fast a human could move.