 The building was a mansion built by Sir Drummond Smith in the late 1790s, which had been lived in by Lord Palmerston when he became Prime Minister in 1855.It was surrounded by a dry moat so they built a makeshift drawbridge to control the entrance.The place quickly became known as Hippydilly.