Is it good practice to always remove highly correlated variables?
(1) No. For example, if you have computer vision problems, then each pixel of the image is a feature. most neighboring pixels are highly correlated.

For example, see the following image from The HASYv2 dataset: