Using entity form display modes
Form modes and form operations are a bit strange. You also need to define a form operation through hook_entity_type_alter() to be able to use a custom form mode.

Which makes the UI to create form modes pretty useless, but ssshh, don't tell anyone :)

For an example, see views_ui_entity_type_build(): $entity_type-&gt;setFormClass($operation, $class_name)