Wise married a second time in November 1840, to Sarah Sergeant, the daughter of U.S. Representative John Sergeant (Whig-Pennsylvania) and Margaretta Watmough of Philadelphia.
After Wise entered Confederate service, he and his family abandoned Rolleston in 1862 as Union troops were taking over Norfolk.