Non-default web port
For HTTP the default port is 80 and for HTTPS it's 443,
obviously.
If port is omitted in the site URL, then the default
one is used.
If you want to use a different port, you must
specify a different port in the URL.


So, the logical conclusion is that the port must be present in the URL. When building URLs to entries, for example, the site URL is fetched to generate the final URL. So, unless that URL has port specified in it, you'll get URLs with the wrong port. That's exactly how it should work.

The best way to approach this, however, would be to use Craft multi-environment settings to set it up. That way you won't have to dig into the config files whenever you deploy your site.