What can I do to ensure I only erase the smaller details without affecting the area around it?
Get an erasing shield/stencil:



An erasing shield is a metal stencil that can mask off the parts of your drawing you want to protect, while exposing only the area you want to erase, so you can erase with more precision.