Vertices being moved by armature despite only moving bone having no weight
Sure enough it was something basic. Rigify used bendy bones for the neck, so I guess with such a sharp angle between the head and first neck bone, that neck bendy bone got bent too much, causing this mess.