Does setting a database to single_user prevent delivery of Service Broker messages?
SINGLE_USER means only a single session can connect to the database, whether that session is SQL Server itself, SQL Server Agent, or some user makes no difference.

If you need to ensure no one can access the database, the only option is to take it offline.  

If you are comfortable with members of the sysadmin or dbcreator roles, or the db_owner fixed database role, having access, you can put the database into RESTRICTED_USER mode.  See this MSDN page for details about that.