Not finding the right tension for my threadless headset
I figured out what was causing the problem. I had used the compression ring in the place of the crown race. I did this because I lost the crown race and the compression ring looks similar.

FYI crown races for threadless headsets aren't standard, if you need to replace one you have to get one that matches your make and model.