pgbouncer bind(): Permission denied
The error message in the log was a red herring, and due to the configuration for Unix sockets (when I only needed TCP). Disabling Unix sockets by setting the unix_socket_dir to an empty string (as per the documentation) made those errors disappear from the logs.

The connection problem was because I needed to specify the password in the pgbouncer.ini file.

The line:

* = host=myremoteIP port=5432 user=postgres


Needed to be:

* = host=myremoteIP port=5432 user=postgres password=mypassword


I can now connect using psql.