Why does Grease Pencil color and stroke move separately when animated?
I do not know whether there is a fast workaround, but I am quite certain that you have to manually input the keyframes for the Fill layer, as you did with the Stroke layer.

So just go to the frame you want the color to 'change position' and recolour accordingly.