Can I force MSDTC to control only within a single database?
Microsoft Distributed Transaction Coordinator is a service that runs on the machine that cannot be configured per database.  The service is either installed, or it isn't.

Since a distributed transaction must use a linked server, you could theoretically use permissions on any linked servers you have configured to allow only your process access.  You could use a DDL Trigger to prevent creation of any further linked servers (or at least to audit their creation).  

Ad-Hoc Distributed Queries can be turned off at the server level using EXEC sp_configure 'Ad Hoc Distributed Queries' - this setting may help prevent the majority of distibuted query attempts, especially if the users don't have the ability to create stored procs.