Entity view mode and form mode schema node_type_example module provides a file that defines form display configuration for a custom node type defined in this module. The file is named core.entity_form_display.node.basic_content_type.default.yml.

The example file clearly sets some field settings but is there a a schema for these config files? How to build my own config file for view mode and form mode?
First, form/user mode and form/user display isn't the same thing.

But the schema for all of them is in in core/config/schema/core.entity.schema.yml.

That said, I'm not sure why you want to build that file? Especially for a node type. Create the node type and the fields, configure the form and view displays in the UI, export it, add the files to your modules, don't. You don't have to build them yourself.