Zooming in on one layer in After Effects
Thats simple, pre-comp what you have right now and create a camera layer. Then make you composition layer 3D. The look of you layer will not change unless you add lights or rotate the camera. Then you can pan around and zoom in your picture with the camera.

Thats how you do every sort of zooming e.g. camera animation in after effects. Animating the scale of layers is just a mess like you already realized.