He married Ellen Douglas Jamison in 1873; they had seven sons and two daughters, and his eldest son, Francis J. Baldwin, became an architect and joined his father's firm.
hiː mˈæɹid ˈɛlən dˈʌɡləz dʒˈeɪmɪsən ɪn wˈʌn θˈaʊzənd ˈeɪthˈʌndɹɪd sˈɛvənti θɹˈiː; ðeɪ hæd sˈɛvən sˈʌnz ænd tˈuː dˈɔːɾɚz, ænd hɪz ˈɛldɪst sˈʌn, fɹˈænsɪs dʒˈeɪ. bˈɔːldwɪn, bɪkˌeɪm ɐn ˈɑːɹkɪtˌɛkt ænd dʒˈɔɪnd hɪz fˈɑːðɚz fˈɜːm.