The Jarvis House is set on the southeast side of Surry Road (Maine State Route 172), a short way south of its junction with United States Route 1, just west of Ellsworth's central business district.
The interior of the house follows a central hall plan.