Change origin of collections
The only way I think you can solve that is by pressing alt-p to clear parent and keep transformation then set the origin to geometry, after that you are free to delete the useless empty and start animating.