It was built by John Sheffield , 1st Duke of Buckingham and Normanby in 1703 . The house which forms the architectural core of the present palace was built for the first Duke of Buckingham and Normanby in 1703 to the design of William Winde .