Force ConcurrentDictionary in a Singleton Registry to Collect Removed Items/Space
Apparently, collections are optimized for performance over memory management. So I'm afraid it is up to us, consumers of the .NET Framework, to implement memory management on collections that grow big and require lots of manipulations.

I would use a scheduled task to periodically make a new instance of the queue, with the values of the previous instance, and clearing the old instance.