Performing data update operations while backing up a large SQL Server database
Essentially, the backup will be of the state of the database when it finishes the data-reading portion of the backup (so all of the data will be backed up), plus whatever amount of transaction log is required to ensure transactional consistency (the start time of the included log is MIN(most recent checkpoint time, oldest active transaction start time)). Paul Randal covers this here (with aid of a diagram, which makes it all so much easier). In your example, A would be committed (or rolled back if a ROLLBACK TRANSACTION was issued instead of a COMMIT) and B would be rolled back (regardless of the end result of that transaction).

(The other reason you try and do backups at a quiet time, aside from I/O contention, is that all of the transaction log generated during a backup normally has to be included with the backup.)

The recovery phase of a database restore takes all the committed transactions from the log included in the backup and applies them to the database, and rolls back all the un-committed transactions. (This is why WITH RECOVERY/WITH NORECOVERY is important. WITH RECOVERY and you can use the database, but you can't apply any further log backups, you need to restore it WITH NORECOVERY in order to roll in log backups. Recovery breaks the log chain by rolling back uncommitted transactions.)

Further reading:


30 SQL Server backup myths
Debunking a couple of myths around full backups
Logging and recovery in SQL Server