If I manually failover my AlwaysOn AG primary replica to the secondary and take the primary database offline will it auto-sync when back online?
This is not possible since database is added to AOAG 
You cannot take the db offline unless you remove it from AG

You can see and verify this yourself simply by adding some "TestDB" to your AG and try to take if offline


  Do I have to wait for the sync to finish before manually failing back over to Database A from Database B?


Yes you have to wait for sync to complete in any scenario, if you do not want to lose any data