James's brother was John Rosamond Johnson, who became a composer.
Over the next 40 years Johnson served in several public capacities, working in education, the diplomatic corps, and civil rights activism.