In Memory OLTP causing OOM Issue in SQL Server even though no data in In Memory OLTP Table
It's a best practice to use a Resource Pool if running EE, that might help in this case. Also, garbage collection does not occur unless there is memory pressure/activity.