 Among the houses in Whitehall Gardens were Montagu House and Pembroke House , a Palladian riverside villa with elaborate interiors . The houses were demolished along with the rest of Whitehall Gardens in 1938 , in what the architectural historian John Harris described as " a monstrous act of vandalism " , to make way for the construction of the new offices of the Board of Trade and Air Ministry . The interior decorations of four of Pembroke House 's rooms were saved after its demolition and were reinstalled in the new government offices , now the Ministry of Defence 's Main Building . 
