Why do we scale down images before feeding them to the network?
Because it saves lots of computation time. Yes, we do loose image details, but it depends on your task whether the details are really important. Let's say your task is to detect and count the total number of circles in an image, So you are only concerned with the shape circle, not its dimension. Hence, reducing image resolution a bit would be more computationally efficient.