This paper introduces a novel methodology for automated detection of buildings from single high-resolution optical images with only visible red, green, and blue bands of data. In particular, we first investigate the shadowevidence to focus on building regions. Then, a novel Markov random field (MRF)-based region growing segmentation technique is proposed. Image is oversegmented into smaller homogeneous regions that can be used to replace the rigid structure of the pixel grid. An iterative classification merging is then applied over this set of regions. At each iteration, regions are classified using a region-level MRF model, then, according to the position of shadows, regions having the same class are merged to produce new regions whose shapes are appropriate to rectangles. The final buildings are determined using a recursive minimum bounding rectangle. The experimental results prove that the proposedmethod is applicable in various areas (high dense urban, suburban, and rural) and is highly robust and reliable.