How can I setup replication with the DNS name instead of the Server Name
I was able to get this working, but with a fairly unsatisfying workaround - so if anyone else can contribute a better solution, please do. I am concerned that this solution will have unknown side-effects.

The broad steps are:


Remove everything to do with replication from the Publisher/Distributor, up to and including the distribution database.
Rename the SQL instance using the code sample pasted below. The new name should be the Machine FQDN (not the DNS FQDN). This is important so that it matches the name of the SSL certificate. Reboot the server (apparently just restarting the SQL instance isn't enough)
Redeploy the replication setup.
Change the EncryptionLevel setting to = 2 for all Distribution Agents.
Check all Distribution Agents are working fine.


Now, when I get to the sp_adddistributor command, it works fine if I use the Machine FQDN, for example:

exec sp_adddistributor @distributor = N'SERVER1.mywindowsdomain.com\DEV', @password = N'12345'


And indeed, EncryptionLevel can be set to 2 successfully for the Distribution Agents.

Known side-effects:


Replication Monitor no longer connects automatically to the Publisher when it is launched from SSMS. I should note that it does work fine when being launched on the Publisher/Distributor server using Windows Auth, but does not connect when I launch it from our Jump Host and try to connect as 'sa'. To make it connect in this situation I have to add the Publisher, and then subsequently edit the connection properties.  I suspect that this issue may be somehow complicated by the fact our servers are all Case-Sensitive, but I have no evidence to support this suspicion.
When using Distributor View in Replication Monitor, the right-panel list of Publications is always blank. The Publications do however show up in the tree view on the left. There is no problem using the Publishers View.


Code to rename server:

sp_dropserver 'SERVER1\DEV'
GO
sp_addserver 'SERVER1.mywindowsdomain.com\DEV', local
GO