 Baker served in the Navy prior to becoming a boxer.He was an only child and had only one child himself, Robert Baker, Jr.

After his boxing career, Baker was a foreman for a PennDOT road crew in Penn Hills in the 1980s.He was well regarded for his honesty, humility and fair treatment of his crew.