Always On Setup - OBJECTSTORE_SERVICE_BROKER Memory clerk eating all the memory
You mentioned that you're not explicitly using service broker anymore, but Availability Groups (AGs) use service broker behind the scenes.  This system is called the Usage Control Service (UCS) and is used for queuing and sending groups of log blocks from the primary to the secondaries.

I was able to find at least one bugfix for this system causing memory issues:

FIX: Service Broker UCS task leaks memory in SQL Server 2014 or 2016


  Assume that you have AlwaysOn Availability Groups configured in SQL Server 2014 or 2016. If an application uses the TABLOCK hint, the Service Broker Usage Control Service (UCS) task leaks memory. In this situation, you can see that the number of tasks in the sys.dm_os_tasks function continuously grows. This causes AlwaysOn synchronization to become slow.


Note that this bug was fixed in SQL Server 2016 (RTM) CU2, so unless this is a regression, this isn't the specific bug you're facing.

All that to say, this could be a legitimate bug in the product that's been fixed in a newer version.  If you're having trouble updating to SP2, consider trying to at least updating to the latest CU for SP1 to see if the problem has been resolved there.