Does setting a database to single_user prevent delivery of Service Broker messages? Is it safe to assume that while a database is in single_user mode, the queues are not available for message delivery? My understanding is that the special tables that the queues represent reside in the database context, but it's not clear if message delivery counts as a 'connection' in the usual sense.
SINGLE_USER means only a single session can connect to the database, whether that session is SQL Server itself, SQL Server Agent, or some user makes no difference.

If you need to ensure no one can access the database, the only option is to take it offline.  

If you are comfortable with members of the sysadmin or dbcreator roles, or the db_owner fixed database role, having access, you can put the database into RESTRICTED_USER mode.  See this MSDN page for details about that.