 Some of the external walls of Bridgwater Castle were demolished in the early 1630s by the then owner Henry Harvey, an attorney of the Court of Common Pleas, to built his new house.Harvey had purchased it from George Whitmore who had been granted the ownership by Charles I in 1626.Harvey leased the house to Colonel Wyndham in 1643.