 In 1860, Latrobe married Louisa Sherlock Swann, daughter of Thomas Swann, who was formerly Mayor of Baltimore and Governor of Maryland.They had one son, Thomas, before she died in 1865.Latrobe married Ellen Penrose, the widow of Thomas Swann, Jr., in 1880 and together they had three children