Removing secondary replica from SQL Availability Group causes large log file
I think the question has a wrong assumption.


  So is there a way to keep a single (primary) replica server active in the AG for the databases without removing them but keeping the log file from growing or being stuck in 'AVAILABILITY_REPLICA'?


...


  I'd rather not remove the AG as the listener is actively used and would require reconfiguration of numerous web.configs and other configuration entries. But, I also am slightly worried about the log file growth.


If you have an AG running on a single node, you can remove the database from the AG completely AND still connect to the database via the listener (as long as the clustering part is online either forced or using Windows 2012+ with dynamic quorum).

For a normal connection without any application intent the listener just functions as a dumb alias for the server. It doesn't check something like whether the database you're connecting to is part of its AG or not.