Transactions and form api
Can you use FAPI with transactions: Yes.

But you can't or rather shouldn't use it the way you describe. Transactions only take effect when you commit them, so there really is no point in starting the transaction in the validation. Also what happens if you have a validation error, you won't commit the transaction. I believe that starting a transaction will effect other incoming queries as well.

Transactions are only good if you want to do multiple queries at once, and do all or nothing. Like insert data in main table and revision table and then update the revision number in main table.

What are you trying to do that makes you think that you should make transaction in such a way?