How to enable field_ui "Manage display" for a custom entity?
I'm pretty sure your view display configuration there is invalid, there is no "file" formatter. There are many formatters for file fields but none is called just file, there is file_link, file_uri, file_default and many more, you have to pick the one you want.

Granted, the error handling there could be better. Feel free to open a core issue about that.