Particle System - Hair: size jittering - messes up scene
I turned out that I had a Force Field and Wind in my Scene that were supposed to only affect the Smoke, somehow that also affected the Palm Particle System.
I deleted the Force Field and Wind and since I had baked the smoke animation it stayed the same and the palms now don't wiggle anymore.