Over-voltage on a current regulator
Let's say I have a current regulator, or a LED driver, set at 4V that can sink 20mA.


That means your power supply can supply up to 4 V at up to 20 mA.  It will limit at whichever maximum it reaches first.



Figure 1. IV curves for various LEDs with the 4 V, 20 mA limit shown in the shaded area. Source: IV curves.


  If I connect a white LED rated 20mA at 4V to it, then everything is fine.


Correct. With the one shown on my graph (which may not be very accurate) the 'W' (white) curve crosses the power-supply boundary at 3.0 V and 20 mA. So the current limit is reached first and the LED will be current limited.


  But what happens if I replace, in the same setup, the LED with a red one which works at 2 V and also 20 mA? 


Figure 1 gives us the answer to that too. The red curve crosses the power-supply boundary at 1.75 V and 20 mA. Again the current limit is reached first.


  Will it work? Will the regulator limit the current to 20mA and adjust the voltage accordingly? 


Yes.



To help further, let's say you had only 2.5 V available and limited the current to 50 mA max.



Figure 2. IV curves for various LEDs with 2.5 V and 50 mA max limits.

Now what happens?


The IR (infrared), red and orange will get 50 mA but all the others will get less than 50 mA because the voltage limit of the power supply has been reached.




Figure 3. A lab PSU with independent voltage and current limit settings is quite instructive in developing an understanding. Random image source: Circuit Specialists.

If you can get the use of a lab power supply to play with for an hour and a bunch of LEDs you can plot your own IV curves and develop a good understanding of how all of this works. If you want accurate current readings you might need to slip your multimeter in series on mA range as most of the PSU meters give only 1 mA resolution where you might want 0.1 mA resolution at low currents.