Customize "view modes" and "form modes" based on role
Nothing in core uses form modes - it's an API waiting for contrib to make use of.
This, and the question of when to manipulate form fields and when form modes, is discussed at:
https://www.drupal.org/node/2530086.

There Berdir says "Showing/Hiding fields but keep them otherwise the same is IMHO a perfect use case for hook_entity_field_access() not switching out the form display. I'd only try to do something like [switching form  mode] when you want to show different widgets based on the user roles."

In general, managing display differently for different roles is easy, managing forms differently is more difficult.

Alternatives to the field permissions module for managing display (NOT forms):

The classic answer might be that this is a job for the Panels family, but it's mostly not quite ready for D8.  Page manager is however quite usable and does much of what you want. For specifically created pages, it can control the content based on role. What it won't do is take over the rendering of all nodes of a certain content type.

Display suite is D8 ready, and allows you to show fields based on permissions by using token fields. See https://swsblog.stanford.edu/blog/using-display-suite-provide-field-level-permissions for a description of the procedure in D7, which is only cosmetically changed for D8.