Configuration syncronization not importing everything
It appears as though the issue I had relates to this issue: Display mode configurations don't get updated with new fields https://www.drupal.org/project/drupal/issues/2915036

I haven't tried the patch yet, but after saving the individual display modes the fields then get put into the export and sync properly.