The Amory House is located on the grounds of the former Amory country estate, a large tract of land rising on the northern slope of Mount Monadnock between Old Troy Road to the west and the mountain's Pumpelly Ridge to the east. It is accessed via a private road that snakes across the historic estate. The house is a rambling 2-1/2 story wood frame structure, with multiple gambrel roof gables and a clapboarded exterior. Some of the gables are adorned with half-round three-part windows, while one has an oriel window at its center. A screened porch with round columns provides a fine view to the west. A carriage house with similar styling stands nearby.