The only catch is that we store raw FAggregator*. This should only be used in scopes where aggreagtors are not deleted. There is currently no place that does. If we find to, we could add additional safety checks.