Heritage and youth. 1  Robert Edward Lee was the fourth child of Colonel Henry Lee and Ann Hill Carter. On both sides, his family had produced many of the dominant figures in the ruling class of Virginia. 2  With all his aristocratic connections, Robert lacked the advantages of wealth.