Programmatically Render a Node Edit form with a custom Form Display Mode
The error message says you did not specify a form class, but what it really wants to say is, that you did not specify a form operation with the name of the form mode:

mymodule.module:

function mymodule_entity_type_build(array &amp;$entity_types) {
  $entity_types['node']-&gt;setFormClass('my_custom_form_display_mode_id', 'Drupal\node\NodeForm');
}


The form operation needs a form class. This can be the same as for default, which you find in the handlers section of the annotations in the Node class.