Using HDR Image makes real world scale architectual model float
The HDRI is rendered at infinite distance. When trees appear too large then this is because the panorama was taken at a close distance to the trees. What you could do is either set the background to transparent and composite with one of the backplates provided with the HDRI (this allows some cheating with the scaling during compositing) or chose an HDRI with realistic dimension that depicts a place with enough free space to actually build a house. For example this one.