James was the son of Major W.C. James of the 16th Lancers and his wife Effie, daughter of the painter John Everett Millais.
As a child, James sat as a subject for several paintings by his grandfather, Millais.