Bike fit: Adjusting stem angle vs. height
Assuming you've got a stem of about 100 mm, changing the angle by 2° will lower the handlebars by about 3.5 mm. If you want more drop, remove the spacer first. Obviously the difference between the two is tiny, and probably not perceptible.