The prison is built around Erlestoke House, a country house built by Joshua Smith, Member of Parliament for Devizes, between 1780 and 1810.
A single-storey lodge from the early 19th century stands at the roadside entrance to the grounds of the house.