How is this extremely high-resolution stepping achieved?
If you look at the notes on the videos, you'll see that he is microstepping. He explicitly states that he is driving the stepper with sine waves. The simplest way to do this is with the PWM outputs feeding the amplitude control of the two stepper drivers. 

Begin by noting that 1.8 degrees divided by .04 degrees is ~45, and since a standard stepper quadrature waveform takes 4 steps to complete a cycle, this implies a microstep resolution of 4 x 45, or 180 microsteps per cycle. In fact, it's pretty clear that he is using 64 microsteps nominal.  

If n is the step count per excitation revolution (that is, microsteps per every 4 nominal steps) , let n = 0 to 255. For each successive n, find the step angle A = 360/n. Then find $$ X = 127\sin(A) +128$$ and $$ Y = 127\cos(A) +128$$ and use the analogWrite function to produce the PWM versions of these quantities. When applied to the control inputs to standard 1.8 degree steppers, you'll get a nominal step size of $$S = \frac{4\times{1.8}}{256} =.028\text{degrees}$$

Also note that, if you look closely at the videos, the step-to-step distance is not constant, and that is only to be expected.