Issues setting up a publisher for transactional replication on a SQL Server with a custom instance
I'm afraid you are not going to make it work. Biztalk uses internally distributed transactions and thus, DTC.
Replication procedures use save points.
These 2 things are not compatible as far as I could read.

References:


https://feedback.azure.com/forums/908035-sql-server/suggestions/33422404-replication-and-availability-groups-with-per-datab
https://am2.co/2018/02/replication-availability-groups-2/
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/466202d8-93f3-41c2-9cc8-a0b7eb594687/cannot-promote-the-transaction-to-a-distributed-transaction-because-there-is-an-active-save-point-in?forum=transactsql