Form #states only work after change, but not on initial load
My given code works correctly. However, there is a bug in Paragraphs that instantly re-shows hidden fields from form API, see:

https://www.drupal.org/project/paragraphs/issues/2868155#comment-12493576
https://www.drupal.org/project/paragraphs/issues/2946856

Applying the patch from issue #2946856 fixed it for me.