How to remove read-only routing URL
There is no method for deleting a read only routing URL for a given replica. Your only options are to change the URL to something else (just putting a different URL in the READ_ONLY_ROUTING_URL = N'tcp://newRoutingString' or dropping the replica entirely from the AG and putting it back in again (less than ideal I know).

If you adjust the replica to SECONDARY_ROLE(ALLOW_CONNECTIONS = NONE) then you can at least prevent any potential attempts to hit that URL.