Software need db owner permission to master database (sql2016)
I have seen software that requires a sysadmin server role membership, at least during installation. It might be that the software creates dbo-owned objects in master and db_owner is the easiest way to allow that without resorting to sysadmin role membership. Although it might be possible to provide more granular permissions, application code changes might be needed and one cannot the grant the minimal permissions necessary without knowing specifically what the software does and the vendor would need to support such changes.

A separate instance/machine (including in Azure) can help mitigate security concerns. SQL Server Express might be an option to reduce costs if you can work within the resource limitations but be aware your HA options will be limited.