The house was built in 1875-1876 by William Watts Sherman, of the banking firm Duncan, Sherman, & Co. of New York, and his first wife Annie Derby Rodgers Wetmore (daughter of William Shepard Wetmore of the nearby Chateau-sur-Mer).
The original house was 2-1/2 stories in height and basically rectangular, about in dimensions, with porte-cochere on the east facade, and two principal entrances on the west.