Why have both a postmaster and postgres executable?
The postmaster symlink appears to be historical. I didn't even know it existed until you pointed it out.

The other two are a typical pattern for UNIX daemon applications. postgres provides the server functionality, and pg_ctl provides control client for it.

It would be possible to bundle pg_ctl's functionality into postgres, but doing so would mean postgres would have to run in two modes: as a DB server, or as a client to connect to a running DB server. That's a little bit unpleasant architecturally, and would add what's essentially "dead code" (unreachable and unusable) to running PostgreSQL server instances.

Similarly, it'd be possible to remove the ability to launch a PostgreSQL server from the postgres executable and require you to do it via pg_ctl, but that'd be cumbersome, and would really annoy people who manage distribution init scripts. They usually want to bypass application startup programs and launch the daemon directly, so they can prevent it from detaching to run in the background, trap exit codes, capture stderr, etc.

Essentially: postgres is the low level server program. pg_ctl is a user tool for starting/stopping the server and other management actions. postmaster is an obsolete alias for postgres.

In general you use distribution service management (service, systemctl, update-rc.d, etc) if you installed PostgreSQL from packages, or pg_ctl if you installed from source or 3rd party binary installers that don't integrate with the operating system services system.