Kingston was born in Adelaide, the son of Sir George Kingston, a Protestant Irish-born surveyor, architect and landowner in the early days of British settlement in South Australia and later a member of the first Parliament of South Australia.
Kingston and his older brother Strickland Gough "Pat" Kingston (1848 - 3 October 1897) formed a business partnership Kingston & Kingston in 1879 which they dissolved in July 1884.