Adding a database with Service Broker to an existing Availability Group on a new replica
Both SET TRUSTWORTHY ON and ENABLE_BROKER are database settings. When you backup and restore a database, these settings will be included in the bak-file and can be restored on the new SQL instance.
Probably, you'll need to make some server level configurations though, such as a service broker endpoint as described here

Edit: Just to clarify, when restoring the backup you don't need to specify the ENABLE_BROKER option. The broker settings will be synced from the primary replica as soon as you join the database to the AG. You only need to restore the database using NORECOVERY.

On a side note, when you add your SQL Server 2017 instance to the AG and failover to it you won't be able to fail back. This leaves your AG with a single point of failure (the new primary replica). 
Consider upgrading all your secondary replicas before failing over to your new instance. After the failover, upgrade your (previous) primary replica. More information about rolling upgrades can be found here