Can I scale a mesh with baked physics?
Here a workaround if someone else encounters this issue in the future:

Simply bake your animation and export it as albemic file (.abc). The animation playback will be slower, but as you import the albemic file you will find a transform cache in the constraints tab. In the transform cache you can simply scale and offset your animation to your likings.