Using CNN to identify buildings from aerial images
You need to know the basic mechanics of CNN , you can't simply finetune a pretrained net (VGGNET) everytime. VGGnet is trained on imagenet dataset so most of the features in it are not so relevant for your task (recognising buildings from aerial view). so it is much better to train a cnn from scratch. You can't simply say that it fails to classify residential , non-residential . It is best to find what could be done by trying,

My issue is that non-residential buildings are mostly larger than residential houses, now I want to use this property to distinguish between residential and non residential.

Most of the times it is better to observe the features recognised by the net and then come with an approach , rather than enforcing your views.

You may find this helpful https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Thesis.pdf