Robert Latham Owen was born to Dr. William Owen of Lynchburg, Virginia and his wife Jane Latham Owen.
The family lived in Lynchburg's best-known mansion, Point of Honor until 1870, and Dr. Owen Sr. operated the Vista Acres plantation in Campbell County.