Why does the forward voltage drop in a diode vary slightly when there is a change in the diode current?
Diodes conduct a current at any voltage across them. It's a continuous curve. However, it's not a straight line as it would be for a resistor.

Here are some voltage/current measurements I made a while back



Because we're usually interested in 'sensible' values of current, like 0.1mA to 1mA, we often model a diode as a fixed voltage drop. As you can see, over that range it doesn't change much, so it's a good engineering approximation. Notes: How lousy a 3V zener is as a constant voltage reference, compared to all the other non-references. A 1N400x leaks less current at low voltage than a 1N4148, say for protecting your +/-200mV meter input with shunt diodes.

Unfortunately, why is a question that, if you're not careful, can go down the rabbit hole of why, explanation, so why explanation, deeper explanation, and so on. Ultimately, all explanations that don't ground in your intuition are what, not why. For instance, why don't we fall through the floor? If your intuition is that atoms are hard billiard balls that stick together, then why is that stuff is made of atoms. If your intuition is that atoms are 99.999% empty space, then there's more explaining to do. Unfortunately, it's a bit tricky to intuit quantum mechanics, so if you're persistent, all why's are going to end up as 'well QM says what', and why QM?

It may be thought this answer doesn't address 'why?'. It does, because the answer is 'it does'. When we do the experiment, that's what happens. Then we construct theories to understand what the experiment is telling us about nature. If we model the device as having bandgaps, then we gain some predictive power, and say that model is useful, or even (extrapolating wildly) right or true.