Concern that having SSH/OpenSSH running with the default configuration (/etc/ssh/sshd_config) exposes an attack vector increasing the risk of Ubuntu being compromised.
The more externally available services running on a system, the greater the risk of attracting unwanted visitors. The default configuration of SSH is fairly flexible in order to accommodate most users, but the universal flexibility comes at the cost of security.