SQL Server: track database changes
You should separate the tests into tests that can execute inside a transaction and rollback and tests that require a full restore. Execute accordingly :)

If you have any DDL in the test then don't bother with transactions, the only safe option is to restore. Most single threaded DML tests will work if wrapped in a transaction and rolled back, but not all.

For speeding up restore consider snapshots, see Revert a Database to a Database Snapshot but keep in mind that snapshot feature requires Enterprise edition (or whatever its current name for 'Enterprise').