Is there a way to disable curve normals by default?
You're right: saving to startup scene doesn't make the change permanent. A workaround I use is changing the Normal Sizes on the same display tab to something smaller and less intrusive.
Then I go to object mode, delete the curve and save as startup file using CtrlU. The next time I open blender,  the normal size for displaying curves is how I set it.